{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本书配套视频课程：[解剖深度学习原理，从0实现深度学习库](https://ke.qq.com/course/2900371?tuin=ac5537fd) \n",
    "\n",
    "更多代码或学习资料将向购买视频课程或书的学生提供。\n",
    "\n",
    "\n",
    "+ 博客网站：[https://hwdong-net.github.io](https://hwdong-net.github.io)\n",
    "+ youtube频道: [hwdong](http://www.youtube.com/c/hwdong)\n",
    "+ bilibili网站：[hw-dong](https://space.bilibili.com/281453312)\n",
    "\n",
    "## 3.6 softmax回归\n",
    "\n",
    "### 3.6.1 spiral数据集\n",
    "\n",
    "下面的代码生成一个二维平面上的3分类数据集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOydd5gb1bn/P2dX2vWu7XXBFWNjwAabZgOm9xaMKaYEQgs9lEB646bcJDf3Aknuj9wkBIgDhBIIoVcDxvSOCxiMjSs27gV3r9dbdH5/fGeikTTSalfalXZ1Ps+jZ1ejmdGRNHPe83ZjrcXhcDgcpUtZoQfgcDgcjsLiBIHD4XCUOE4QOBwOR4njBIHD4XCUOE4QOBwOR4kTKfQAWkOfPn3s0KFDCz0Mh8Ph6FBMmzZtrbW2b/L2DikIhg4dytSpUws9DIfD4ehQGGMWh213piGHw+EocZwgcDgcjhLHCQKHw+EocZwgcDgcjhLHCQKHw+EocTpk1JAjyHzgaaAS+BrQp7DDcTgcHQ4nCDo0U4BjgQak3P0XMAMYUMhBORyODoYzDXVovgtsBeqBOmAd8NuCjsjhcHQ8nCDo0KxJet4IrCrEQBwORwfGCYIOzXigKvC8GjijQGNxOBwdFecj6NDcCGwAHkQ/5S+Acws6IofD0fFwgqBDEwX+5j0cDoejdTjTkMPhcJQ4ThA4HA5HieMEgcPhcJQ4ThA4HA5HieOcxQWnAZgAzALGAJfg5LPD4WhP8iIIjDFjgT8C5cCd1tqbk17/EXBh4D1HAn2tteuMMYuAzUAT0GitHZOPMXUMYsApwNtALdAVeAW4v5CDcjgcJUbOS09jTDnwF+BkYE/gfGPMnsF9rLW/t9aOttaOBv4DeN1auy6wy7He6yUkBAA+At5BQgBULuIRYFkzx8WQ7HQ4HI7cyYcN4iBgvrV2obW2HngIpbym43zgn3l4307AVqREBYl429NxP9IcegPDgIVtMzSHw1Ey5EMQDAKWBJ4v9balYIypBsYCjwU2W2CSMWaaMeaqPIynA7EfKhHh/wwRYEdg1zT7fwJcjQrMNSIhcHIbj9HhcHR28iEITMg2m2bf04C3k8xCh1tr90cz2nXGmKNC38SYq4wxU40xU9esSS621lHphvwDhwL9UUnp10nvuplC4k9mUT+CujYco8Ph6OzkQxAsBQYHnu8ELE+z73kkmYWstcu9v6uBJ5CpKQVr7QRr7Rhr7Zi+ffvmPOj2xwKLgdloNe9TiWTpNmAFsDbDOQaRKnervXM4HA5H68iHIJgCDDfG7GKMqUCT/dPJOxljegBHA08FtnU1xnT3/we+AszMw5iKjCZUDG4EcCDyqa/0th8DvAtsQh/9aGB9mvN8xXt0A7ojIXAf4UqZo8NhY9DY2Px+DkeeyTl81FrbaIy5HngReT7vttZ+aoy5xnv9Dm/XM4FJ1tqgJ7Q/8IQxxh/Lg9baF3IdU/ExAZhI3ITzOXAlcCvSApoC+zYB04ATQs5jgEeBV73jDgR2b5shO9JjLZg8C99334GXX4ZYDAYPhvPOg6rq/L6Hw5GGvOQRWGsnopkuuO2OpOf3APckbVsIjMrHGIqbacRDREGmoY/Qqj55BdgE1GQ4lwGOy+voHFmyeTP86yFYvhy6dIEzz4TheRDEC+bDq69Ck7cgWLoUnngCLrgw83EOR55wKaztwt4kNpApR2aiHYBrUTgo3t+j0ErfUXQ8+ICEgLWwbRs8/Ah8mcmnkyWLF0NDQ/x5LAZffJH7eR2OLHElJtqF64DnkC+gHGkCd3uv/QFN/tOB3YCLcTb/IqSxAVatkhDwMcCSJbBDn9zO3b07RCKJ/oGuXdPv73DkGScIsmY7qqLxMaoJdD3Zf31R5EL5GEUHjUKOXtBscpb3cBQtsVjqNmOgOg92/P32g+nTYd26uKAZnykn0+HIL04QZEUMOAn4AE3kTwCveX+zXb2XAaPbYnCOtsbG4B//SNxmDAwcCMOG537+SBSuvBLmzoPtdbDzUOjVK/fzFopttVBfDzU1YJz1uSPgBEFWfAxMRUIA5PidhPIChhZoTI6caGqETz+FrbUwdGcYuGP6fdetg5UrE81CZWVw4on6mw/KIzByZH7O1VrWrIFpU/U5R4/O/J2EYS288ILOYQz06AGXXALdMwU/OIoBJwiyoo5Uv3oZLqO3g9LUCHffrYkvFtOqdfx42Hvv8P3D8uTLyzvXanfVSrjrrrjT+sMP4aKLYMjO2Z9j9iz4cHo8+mndOnj8cbjk0rwP15FfOtGV3JaMAnoRLxAXRcnUwwo2IkcOzJolIdDQoEmrsQGeezb9/jv0hj59NPmDtICuXaFvjk7iQrBhPSxcCBs3JG5/883EyKWGBoW0toRlyxPPYa00KUfR4zSCrKhCNYG+AXyGBMPfcF9fB6W2NtHMA7B9e/pEMVOmVe1Lk+DTWbLjb94Md9wBl16mqJ9C8tlsmDxZUUejRsExx4RrK1M+gEmTJNCammDcONhvf71WX5+6f9i2TOzQG6LRRGHQo0fLzuEoCE4jyJqdgOdRVvCTQHvUO9rivecknBkqj+w8NPF5WRkMGZI5W7iyUs7hpkYJjMZGWL8ennoyvs+sWfDkE5qUazOVEs8jixbBY4/Bl1/Cxo3w7rvw+uup+23aJCHQ2Cih19gIEyfGxzl6P03iPtEo7L9/y8YyerSyoqNRfV9dusBZLhquI+CWtGl5DbgUWIOqgz4EtKcpYAWqv7fRe94fRS114GiSYmHAAE1QzzyjSXHIEDjn3PT7WwuTX4J33kndvmqV/n/7LU3ADQ0SLB/PgG9ep8mwLZn5SWL+QUMDzJgBxxybuN/GDdIEgvuWl8PGTVDdFfbcE+q3y0QUi8HBh8D+B7RsLGXl8issWwbb62HHHaGqqvnjHAXHCYJQFqIWkn5ZiDeAU4H32nEM30WF6fwbdzvwC1SfyJEzI/fUIxtmfARTpqRuNwZ699b/vhAATaR1ddIQWrqqToe1Mu18/LFW28cfDzsOgooKjSNo6gqu7H16907NhYjFoFfP+PPR++kRZMMGff6mJjnT+/XPPM7Fi2HlKpmJ2loIOvKGEwShvEFifkADKrK6nfYr+TyfxDpE9cC8dnrvTsayZXr0qIHdd295tM+CBYl2b5+uXeGMM/V/U1Pia7GYnND54q03Ex26S5bAN66Cgw5WMlp9vYRBJAInhBQs7NoNzjxLUTxlnuA45xzokmHFvm4dTPhr/NzvvQdf/zoMHpK439q1Mj3NmwvTpumzl5XBXnvB+DPy9x042gwnCELpRWqiWASoaMcxHAHMIu4bqAaObMf37yRMm6rYdtDktMsu8LXzWlY9tEePuIPVp39/uPwKrchBppXPPoubXsrKYHgWyWZz52oVXVMj7SFsNQ/wwQepUT0fz4DjT4BrroWpUzRh7713+pDPkSPhhz+EzZv0fhXNLGreelOms+B7vvSSPrfP5Mnw/nv6vEHnclOT8jQOPbR5LcJRcJwgCGUc6hkwE03EXYCbad8aQDejJjavo0D2scBP2vH9OwFNTfD884kT+Oefy8G6yy7Zn+fwIzSp1XqmQmPgrLPjQgC08u3yAsybL01h3Djo1Tvzed96E954QxNsJAIffagM4/KQ2zJZcBkTT2br2RNOODG7z1JZCZVZBDps3gQrVqRuDwqGZcvgg/fT91AoK1PCXnNYqzLc770HGDjsMDjkkOaPWbJE0VsDB8ZNdI5WUUKCYB5wNgr/3Al4GNUMCiMKvIkaxa9Aq/Nj8jSOpcj+vwA1obmJxMqkPlWoPtGXKH/BOYlbTH19eDLY1hZG9FRVwbXXqgREUxPsumtqyGgkAqecmv05YzHF6ft2+8ZGmWLmzgvPMD78CHh5clwriEZhVBuVLFm7Bu68M9UcFo3CPvvEn69f17xmNSALbWD6NHjttfj7vfKy/Auj03w+axWtNWuWhE1TkwRzoTOzOzAlIgga0ES+As0Mn6PGLwuBdCuJCuCKNK+1lk2oxPQa1Hdgjvd4Ps3+hvaNVCpirIUtm2Xf79Yt87712xUq+cUXUF6mWkG+M9VaJYJ9sVgTfJ++2ZmJKirTZx63hlhTai5DYyO88bom4sMOS9QMDj4YqrrAJ59AZRc4+ui2WwVPmpS48gcJukMPhcMPj2/r1y/VAe1/l927KxIrm+Y6M2aEm72CgmDTJkVIxSzUdJcQCB7zxOMw4j86V7Z3O1IigmARmoSTl4czUMP49uJ1YCvxjmTbgJeBDUDPdAc56rfDA4FeAMOGaZLxM32DWKsCccuXx01CZWXaXlUFxx0H99yjCSwWk23/jDPz33GsOSJRxdwvWxYfZyymTNy1a+U3uPCixHHtO0qPtmbz5tRtQ4fCsUkNkfr1V70lP0mtrAwu+rrCRlvyfVaGRBetXq2ciIMPVn7EhL9q4veT/pLP39QE9Q0yfTlaTIkIgl4o6iZIA+m1gbYiZOLC4vL6mmHSpMQJc+FCxe0fdXTqvlu2wPIViX6BaFSCY7fd4E9/VGinz8yZsMcICYT25rzz4cknYfGixBV4Y6MEwYYNhalCusceEka+7T8aVbRVGAcdDPvuC1u2Qs8eEnAt5bjj9B0EV/hbt8Krr8BqL0+jLimhMlkQdO/uhEAOlMgM1Af4AeoAFvX+ngHs24pzNQE/A3ZFpSZeasGxxwD9iEcfVQNfJXNrSgdLlyZO7A0N6Tt4lRlSND9r49rD+vWJr8ViMHt23obaIqqq4PzztYquSIpIMyY1JLW9OOoolaooL5dJ6MCDYEyGrnldqlSLqTVCAOTs/cZVsvEHq7n6yXFbtqQe0727xlderuqmF13Uuvd2AHnSCIwxY1HXlnLgTmvtzUmvHwM8hYzzAI9ba/8rm2Pzx41oIp6BisWdQeuigG4AbiOebHYGMvmkczwHqUbZwf+J8gSOBn7UijGUGDv0iVcKBd38ffvCui/h80VaCY4YoUmrazeFbc6frxVtebnCPwfvpGOTk69AZSPyTVOj3j/M7JHMgAGKNGps1GcsL5f9v1CRMGXlcOppcee3MfqulyzVOHfbNf+2+L59Ye99lLMRDEM1RgIi2M4zGoUxY+Cww1X3qaq6/U17nYycBYExphz4C3AiComZYox52lo7K2nXN621p7by2DzxFe+RC/eT2Ii+FniU7AQByBzlsoNbxNixsGyp+gSDJvahu6jom89bb8IVV2qS+Oo58M7bCi/s00eOVd/x2ru36vL4GAO75bmK7GuvKvkLNMlfeFHmTmaRiGLzJz4ngTdwIJw8Ln+9DlqLP7nOnQuPPhJ/PmQIXHBB/oXBLrvou/B9AZGItu23v377t9/W9v33hyOO0PtXu5ae+SAfGsFBwHxr7UIAY8xDwHiUDdWWxxaI5BVehHjzeUeb0L07XHedVqRlZVrd33prok153Tr46CM48ECtqI88Kvxc55wLf79bK29rNantl8cwzM8+U00iX3tZuVKF6C64MPNx3brBuV/L3zjyyZNPJH7XixfDnDkwIs/hmlVV8I1vKPdjwwY5qE88EebNk89iv/0UudQtQ7XXpkZYs1ZCZIcdnKaQJfkQBIOAJYHnS4GDQ/Y71BgzA1gO/NBa+2kLjsUYcxVwFcCQIUPCdmkB61HEzkBabh76H28YtcQb0V+e43gczRKtUPy+j68d+DQ0wNYQW3Iy/fvDd76jqKKKSthpUH5Xtku+SJw0YzH5ODoqNpbqqLUWNoVEFuWDnr3g/Aviz6dNhRdfjBfz++gjFfPrGrL42rxZQn7rVo1x6FA47zyZuhwZyccdEDaTJsdpTgd2ttaOAv6M6jhne6w2WjvBWjvGWjumb9/WloC2wDeBAcBuqIfwmhae40I0/CuA76E2loNaOR5HRtas1mpw08b4toZ6dcHqXpNoOolGdcNP+QCWL8t83qpqmYMGD86/eaNHD61GgxS6X0EumDLlCySvrHfaqX3e/5VXEov5bd+uwnthPPO0Qk3r63XMokUqzeFolnxoBEtRuy6fndCq/99YazcF/p9ojLnNGNMnm2Pzy4PAfcRDSWcDlwEZulOFcqL3cLQZL09WyYHyck0AZ56l8M8Jf1VyUVNTPKbcd66+9WbcETx2LByQrd8mj+y/vyJd1q6Nb+vohdfOv0C5Geu+1Pc9bpxyBdqDlhTzW706McGtocF1SMuSfAiCKcBwY8wuwDLgPOCC4A7GmAHAKmutNcYchDSRL1EmVcZj88u7KKHLpwE1pXcUFStXwPteDRs/lv3xx1RgbePGxNo23bvDmWfCQw8lmmSef14llcOSzkBJasuWa/U+aMf8mQ8iUTl+F3rRL0OGdPzm7T16yEdTv12aV3tm7+69j8pg+795JKI8hzD69tUiwV8MRCJy1juaJWdBYK1tNMZcjwrjlAN3W2s/NcZc471+BwqWv9YY04iM8+dZay0QemyuY0rP7qiGj29fNkCwUqNFpaZdHfWCsn5DeMRMshAAmQpqQwqbWTRxhZU42LAB7rozHp3St69aUaar/NlSystheJoErI5Mc9VK24KTT4bKCpWU6NIFxp6cvprp6afD3XfHW5EOGaIciJZQvx0wqXkdnZy85BFYaycCE5O23RH4/1bSxEyGHdt2XA38C9n1y9DHv9d77QngYuQEHuYNabf2GZYjkX79Uk0C0Sjssbuch/7Kv7xcDuRBgxJzA4xR74F0tfaffSbuUAR1GXv3nfBMZUdhKS+HE7+iR3N0r4HrrpdvKRJV6HC2UUONjQqRnef1/Bg5Ul3sSsTRXCKZxT6VqOnMRCQQ5gMjUGXSC1GP4Jj3/Cuk8Vs72poddlAyU3lEAqBLF7jwQuUOnHKKnvtC4IwzFWly7rnxtoh9+sDXL04/CXz5ZaLgaGxU/L6j4xOJwMAdpeW1JHT01VeUzObXoJo7F956q+3GWWSUSK0h0AT/B+Bx1P/3JuKlnaeQ+FVYFNW6CejhbVsAvIrKQZyOMx+1MaNHw54jVc++pns8IWzU6PDyy8OGw49/Eu+OlYkdB8mW7DsWo1HYaXDmYxydm0WLUns/f76oZLTEEhIEN6Ak5lrkG3gZ+BQFKg1EgiJIOeCXO34DNavxC8QNRaUiXGPuFmNjsHqNyjD365/emQuySbfULp1NNu6ppygCZu1aaQa7765ENEfp0quXGvH4mmJZGfQunR4gJSQIbideGsJ3Cj8GfAfVIBoLvBB4/Xbi1UKvJDHaaAFwJ/CtNh1xp6OhAe6/TyF91upm230POPoo9QUIw8aAkLLDuVBVDVddJa2gPNJ8fwNH5+crJylj2q9z1KUKjvVK1G+v0zVbWQn9B3TKbOUSEgTJP54JbDPAI6iS6FLUPCbQiYm1iYdSh5rcOFrEm29q1RVUwWd+AnPnwFVXyzfgE2uCZ56JJw8deCCcdFL+QhdNGfRwPSAcHjU1cP31MgctmK8M5ltuUd2ndeu0cInF5Jc692uFrwOVZzrXp8nIdaj6p08T4HdbmgdciyKIBpEoBACOIrFxfRXt29Cmk7BqZXh/2/p6mJqUz/Ha6+oV4Dvvpk93WaKOtqWyi7rAzZih69RalSKpq1OYckODemF8kiazuQNTQhrBjag0xFzkD4ghp+9E4EgUMWS9fe4GggXA7gXOQuWmo8jR7DKLW8zAgbqRwoRB8rb581Kdd3PnwcHNNDV3OFrKmtVqfrR1q8w/yb2agzQ0JFav7SSUkCBYj3oU+07hJmAz8Gtk//fDCWtRv4CgIOiBnMuNyG/Q+WyEbU6sSc1OPv9cRdiCpQAiURid1IKxe/e4LwFkl63pwDV7HMXJxg1w551x30BZmR7JvZh9olH5CToZJSQIygjPC2ggNWIoua2lTwl9Xflk8WL454PxJLGTT1ansHnzFBV0/HEwKKmI2UknqQuZf0y0IrVnrsORK599lpi86AuAior4/5WVEhSxmHpGF6KtaRtTQjNbT9RN7Dm06q9AbSN/ALxCPKKoGvhGIQbYOamvhwcfSOw6NWmSSkFnyhbtvYOyROfOVfvJPfYILxfhcOSCKUuNAopE4LTT5RsYOlShpRvWSyBk6oXQgSkhQQCqPnozsvXvDvwGdQx7AvgpEgaX4tpH5pEN61O3lZUphr9rM2Gb3bqpmqfD0Vbsuac6yvnVbKNRNb/Ze+/E/XboU5jxtRMlJggiwM+9R5B8tLB0hNKte2rdoKYm2LARnr9dq65+/ZTtO3in/LeNdDggfcZ5t24KXX7tVdiyRTWG9j+g/cdXYEpMEDjanepqOGksTHpRWcRNTWo5+Nyz8eiMjRvlL4hG1Yu2RNL6He3AJx/Ds8/KNLnjjuqtkJxA2LOnalaVME4QOPLP3Dnw2mvQFIODDoQxY9SEfO0aNZCZOTM8RK+hAV5/HQ49LH8loR2ly4oV8PQz8UY2K1fCvx6CK64s7LiKECcIHPnl84XwyCPxHIAXX9TfA8aoKihAdE76ED1jtHpzgsCRK198QUKkYCwGy5bFO9s5/k0JZRY72oVp01ITwaZMSdxnv/0UgZF8MxqjCI1qFx3kyANdu6ZeY2HXncMJAkeeSW7cDvINfLkWViyXmt6tO1xzraIzdt9dNttIRA3RM/URcDhawp4jlc0ejer6ikTg9PGFHlVR4kxDobwEzEJNa04q8Fg6GIceqraCvg8gGlWXpzvukDmoshIuv1zNZLLpOuVwtJaycrjkEpgzR30thgxO3+ayxMmLIDDGjAX+iOov3GmtvTnp9QuBn3hPtwDXWmtneK8tQrUemoBGa+2YfIyp9XwfmIDKSUSAy4E/FXREHYr+A+CKK+C992QiqqmRacg3FzU0wOOPq8G7w9HWlJXDyM6XCZxvchYExphy1PHlRFTDeYox5mlr7azAbp8DR1tr1xtjTkYz7cGB14+11ibXei4AX6A+BHXe8+3A34DvAbsUalAdj/4DYPwZ+v/FFxIjhKyNN4RZvx4a6qFXbwmKqi75KzPtcDiyJh8awUHAfGvtQgBjzEPAeGRbAcBa+05g//dQW7B2pgGYjCb7DSij+Awg2BBlLSo9URfYVuFtd4IggfnzYfUqlYLYY4/0dv3+/WUe8oWBMYoeevRRhZmChIBvNrrgAtc20uFoZ/IhCAahBr8+S0lc7SdzBfB84LkFJhljLPBXa+2EPIwpiTpUanoW8ZpCXVBZiemAP/HsQar/3AAj8z+kjsxLk2TuaWqSI3jPPdMn5IwapWSxuXPjk/0eI+D11xKji2Ix2LYNHngAvv99FZlzOBztQj4EQdhSMKzMJ8aYY5EgOCKw+XBr7XJjTD/gJWPMZ9baN0KOvQq4CmDIkCEtHOIE1J94W2BbHdIS/hP4u7etEhWd25D0UZJKJJQyWzbD++/Hy0bEYnIOH3a4SkX4LFsKL02Gum2w115q+9fQCH37wOTJ6Wu+x2KwYQP07Rf+usPhyDv5MMguJb6kBpl9lifvZIzZFzX6HW+t/XdnB2vtcu/valT97aCwN7HWTrDWjrHWjunbN01/27QsIlEI+DQlDXURsDFpnxhqVhMq20qPbXWpDefLymBbbfz5mjVw772weBGsWqUWlTNmKJQvEpXASJcw1tTUfDE6h8ORV/IhCKYAw40xuxhjKoDzgKeDOxhjhgCPA1+31s4NbO9qjOnu/48qv83Mw5iSOJLENpU+1ahLmU8NihYKsgW4BjW3z9C5qFTo1Uu12oMYI1+Az6xPU5PKpk2LPx+9n3q/RqNxoRKNSkiceKJLKHM42pmcTUPW2kZjzPXAiyh89G5r7afGmGu81+9A9pcdgNuMnIp+mGh/4AlvWwR40Fr7Qq5jSuVMFPnzW+JmngrUx/ibgf36AVcC96CuZT51qHT1ESg46nvexylBIhG45FJ4+F9q2dejB5xzLnSpiu9T5tV4tzZxW/D/r52n6KH6emkBmzZKU3Bx3g5Hu2Os7XgmjzFjxtipyc3Os6IehYT6pod07o3HkSsj2UwE6lncD/kcerRiDCXAxo1w+21q+A1a7R9/vOs33BasBz5G65K9m9nXkX82bFAxu5oaVTctcowx08JytUoss7jCe2TCAGejhvXPk2oqakB338O4TmZp6NEDrr4a3nhDkUB77w177xN/vakRamtVC6asPP15SoHtwCpgAM1fmslMA473/m8AvoqUWVeho3347DN4/LF4AcVRo+CUUws9qlZRYoKgJUxApqBVyE8QpIlw57Pj3/TqHU8qCzJ7ljKLQWam88+HITu379jamqXIGLoMuaC+SfjkPBE4Fymh5SgmoSVtmb9KotL6GFrDnB6+uyOPxGISAsHotxkzYN99YXBLoxoLjxMEaRmA8g5mAz9DyWieqYMIcEqBxlUENNTDa6/DyhUwYCAcc3R2cf8bN8ITT8QdyY2N8Pe/q9Dcrru27Zjbi7XAfkhpbALeAhYDv0vabw1wDvG0FlAa5jIUs5ANS5OeNwDzsjzWouwfg+L8nBbRMrZvT+28Z4w67yXnQ27fDrNn63ofNkyNcIoMl8+fkQpgFNIOqtHdYtCd3slWsdliY3DfffDB+7Bwof7ed5+2N8fqVeEZyP96KH1eQUfjSTS5+3NELfD/gLlJ+81F7qYg21AMXrYMJ3ECjwL7ZnHcVuBolD+5O4rVq8t4hCOZLl1So9tiMRiQFOywrVb+sonPqdzK7bfB8mXtN84scYIgK76HzEPWe0wD/q+gIyoYa9YqNyC4ql+6FG68URnHmQRCj56pqyjQDbQxzDHfgViJIoy/T+qkGgPGIM3AZ3DIfk3e8dkwExiG1ipdvL/fRkFtIG3hLuABVNIxyA1I4NR5j7eBX2f5vg5hDFx0kfxckYjCoMedkpoI+eabsGmTFjqNjYqSmzgxcR9r4e234Le/hZtvUjOnsKZNbYgzDWXFdBJzCLYB7wN3A6uBo4DDCjCuAmBj4av6xkaVnejaVVnGYfTrB6NHJ+YU+MfOmxfvYNbRaAKORWaZdEnotSj+4D+950NQqat/Je33KZqcK1G085+R/+AG4pHOnwGHeOe0SBDcgqKhAWag1JkmpDH8HF3CvbzXPyBRCPmXs6Nl9B8A3/+Bmt5XVaUmSa5fDx98kBhGDdo/yCcfq0WrrxVPm9I8jHQAACAASURBVKoCjO3Yu9tpBFkxkkSZWYWWUd8CfoGWYX8POa4T0revbJzJ2cWgC3n27MzHn3oajDkwdfsrLysvoSOwDUUi+ywGFpC5EoklNQDtQlLzHCNICNwK/AYlvi8BfgQ86O1zB3EhAJrUg5XSv4m0gFpkBloO/D7w+kgSI5QqgV1RINxjpGoQjvSUlSl0NCxTfuJz4X6EYcMSt306K9E02tCgsi3tiBMEWXEHqq1Xg3IQBgOb0J3W6P39dsFG166UlcNllykctEuX1NezyQo+8shUQVJertIUPmvWwEP/hLvvgnffyc4H0dbUoYic7mgCvxqZfSoITzoPzg1VwAlogv8jEh4nI3dTV2/famRxNMA/SHQk1yIzD957Jaf/bETurJHAnKTX6pET2ud/kUbSnfjl/CTKpbwU2As5sx25sX596rZu3eCksYnbulanatntnF3vTENZsSOKHvoI3fXvoSVakFo0K5SAbO1SBWecAevWwYS/xv0F5eVwwonhx3y5Fj6YolaV++yjlVRwtdTUBL0928XGDXDXnfGEtJUrYevW9OduD+pRjuFLxFf+96Pgsh9mOG5vVOn8uyjB3V/J/wJ4B3gVmYdWIOuib1XrnnQeA/jBJpciBdSPYK4E1qFIZ9BdHSUunKpJbLTXB/gE+QkM0haeC3yu7cic9NcMn8vRPEOGKOHMv84jETj88FTt4aijpEk3NMiMVF7e7t37SiyzOF/MRt4/f8kWBQ5FZShKjE2bVFuorg4+/1yroL59Yfx4qPEyr9euhb9NkKMMvBviCHjn7bhAOPLIuE303Xfh5cmJgqKiAv7jp+372Xy+QDb3pUjWBykDRqMooOR0kzKkKP4BNbq7j/hka1C0TnJBlSZ0Wc1COQXbvH2rkW1/Eer19ylxrWCwN8YwylH0869IjDCy3ufxE9E+TDrOoMT5O4CvpTm3IzP12+HBB2GJV6V/n310X4Q1X9q8CWbOlJN45Ej1+WgDXGZxsyxD5aeH03yK50jgUaRLr0eJZw+16eiKlpoaOPBAuPVWRf5YK2fY3XfD9d/SpP/eu3EhANIg5s6Bb39HQqJHjRLQMlHIhvYXEC4E8LbNJNw0FAN8t8cqEn0IFsUZBPkrEhzBc+2M8g2uRn3+ziY1l3EZmrjD1nQV6HINfn0N3nleQsKqG3I4Bx3IFt0Ol6HopANCzu3ITEUlXHqZQkjLytWLIx3da+DQwgWclIAdozks8q7thkIxdkWev+Y4Gd2BtcAk1PGsRFmzViUjfO3SWpWWWO3NdEEh4NPYKHvp0KGpQmCvvSRA/Mk/GoWDQquTtx1vonINN6K4gEwuinrCJ+FKNIkDnEWiY7gamYp8PkCho/65/McKJAB2Q/6FsIT2Lt57hcnKbcAjSdtuAV5GE38tWsv0RdpDMk1ASncQR4uoqg4XArVbZTJtSo4iaH+cRsCTSGff7j1qUSXtlmT2lDjRaGrcs7VxW+jo0arL4kdGRKOw337pz1dTA9+4Cl59Rb6BPfcMjzRqK76DYvBjZFdJJAIMRFnF/v7lKJHsNO/55Sj65xY0uV6OGuT5vE+4sNkOvBJ4nzBiSCF9Hl3OqwKvlSGfQJD3SXRENyCH9RaUZRwM3oqGHO/InVdfgbfflj+gokIVfQsYPu00Aj4h8a6IIR+AI2t694bBSXn1kUjc+bvrbnDmmbrQe/WSL2D4cDmF07HDDvDVc3SDHHhQ+5mGZgF/Q2GXzQmBShRINgg5fn+DJv7vo8n0usC+BtnpN3nn9vMDfAYRviKH+ET8A1LDTSuQ2ehPqArKQd4+5WgS70GiwAE5sIMBXxFghLftbyjCqRIJhxE4H0G++Xyh/GBNTdKWt2xRdn0BcRoBw9Gd4/cfMLhG9S3EmMRGNKDV//TpmsQBRu6px9YtcM898Mbr0hqGD9eEX5bFmsRaRR21ZT/jFWhyzSQEytAEez8SBGPQ5PkD79EazkB5iW947x1Dk3EE9fUDxSNMRuGfW1H6yrHe3/XIlLQMhaSeigTBhSjoLcgNSHv4DF3uvYC/eK+diYLiXkOlrc8hs8us3jt2NupUfhluedkcq1alatAFzqFxgoCvIX36WeJxdyXq+M2FdesSnzc2wuqQYPSnn9G+/o0wfz5MnQIHHZz5/HPnwmOPSsDU1MCFFyk6Kd/sQ+bEsBFowv45qcXFcqEMXYKvoMl8PVqfHIectT6HoqQvn0dJzCuoQ1bNScTbbiRTDbyLKqU0IEdwoK8Q+5JdzaIYEkJTkPB6AAXO3ZfFsaVMr96p4dM12VYabBucIKAMTfyzUJjEvqQGcTuapX9/hY/6DuNoNLxRx6qViauhhgZYntLiOpENG+DRR+I+ho0b4f774HvfCw/Fy4U+qLBscukHnxdRMlZbUIaSzlpCWCCKJf2dbZEZaxkSALs1c/5tKON4A+p9EGx+MxUJE197qvX2/R3Kr3CEs/vu8nvNmhXXhM85J/MxbYwTBID0470KPYiOzRlnyOSzebOEwR4jYPSo1P122EG5B77AiERUgygdc+fApEmppqdt2+RI7pZnof0H4Jk0r5XTdkKgtZyIHNV+871qlHAWkvSN9V57DF3y24GrkKlop5D9a4EDURZ0HdIAdgZuRop0Lal+jXJcq47mMAbOOFPhorW1qlhalUUmcd02LYpqeuQ989gJAkd+6F4D110H69ZDRTSeTJbM6afDXXfJSRaLSWs4OMkstGUzrN+glf9TT6YKAZAgCfZJzhePkRg7EGR0/t8uZ7og08zvUNTzsaRvnPc6+nzBdtx/QcXwXgf2T9r/XpTAFvw+FqGIp2rvvapQtFEMWVV3pviEZbHSv5n+3CtXwGuvQd12LZY++jBuUjrtdDXByRN5EQTGmLGogko5cKe19uak1433+jh0WV1qrZ2ezbGODkRZefoQuPnz4KmntJIfNEjp9+vWwYABurDLvUvxw+kq01teLmGRnPlujLSIsWP1N9+sb4SbfgrHvwJdt8JnI+CZ0+Afl8Ef65HnNF2v6wIlvdUA/53Ffn4jmmS2ANegXIYgawnvU1CLWnSchsxMlwHzkZP6btJHPzmyZ80aJWX65tDFixJff+Zp2GUX6J4fjTjnO8kYU47WFSei/MspxpinrbXB8nkno/Cc4Si24Hbg4CyPLRIs8E/gaRSG8ROgGYnuEGvXwr8eVsQPKOV+yRJN8nPmqMXf1ddISEycKA0gTAsAhaqec45KAOeb+u1w57Fw4BSoaNRPvudncMaTcNc3oCyGYip/j2wq5cjr+lXUkGA3dH2MyP/Y8sEBpE+MWxGy7ThkBgrTkHxlbFdKsrJKmzPjo8zNmsrLtZAqFkGAIpfnW2sXAhhjHkJN94KT+XjgPqvCRu8ZY3oaYwYCQ7M4tki40XvUoq/tn6jgSwlnFGfL4sVJdW4Cq/ymJvkMZs+Kl7dOJwQiUTj55PwKgRXL5YPYtAkaNsF33odyb7b0x1wG8Rl0K8pE/zMK8zmJeN3m+che8gWp7ceKgD1RGYvLSSxjUYkcwckcDtyG8iGC5qRqJABu8PY5LfVQ6lF/hhryG11VKjRXA66pSTk5eSIfgmAQUjp9lqJVf3P7DMryWACMMVehZRhDhhTCCHkT8aVRI8oMehR1H38TmQyOITyMo8SpqsqcENbQILPRuHGp9dvLIzBsN0UHHXSQ1GFQka7163UzdK+Rv2HKB9I0+vRRlcfyiPIVZs2SGWmnwWoTuH27jlm/HjYESgVXbCd75iCBEIxaskgoLCYx5rOIuAglzv8MlbxuQkLgL2n2v8R7vIfMQWUoM/lPyClcDfwY+GXgmEUoxHY9ulXOR5nari9y9ozeT42efK3Ajy6KRnWPjD05ryGn+RAE6Qym2eyTzbHaaO0EdCkyZsyYApRMTV6lxlDHj91R+IVFoRfv48JPk9hjD8X8r16tizhYk8gnFlOLvuNPUOXR8nJtO+ssGDEy8XwffQjPPad9/Jti6hQl6lirSX/uXNh5Z3VD82+mFQH7x9q1qeOsr4QvhsBOX0A01ozZP4ayrpIN4nWomlyRCgLQXf9btLaJkd0scIj3eAGF1gZDRv8bZS/7StAF6NbwZfrDqNLqeXkYe6nQty9cdjm89qoWLqNGwbDhWrj07KmFTB7JhyBYSqLytxO6DLLZpyKLY4uE89AV7d8BEZT98yVxs8FCZFT9n3YfXVFT7jWzmTkTtmyFgQPh/fc0WQcxRpVKrZUWceZZcioH2bJFQiDoR3g2Kd6zsVGZmmvWpDczpeOf58NJL8CuC6F6K1Q0gUm37thOYiYWxJMB/oK8qKDr4zMkWUZQNN7UMu/xKtIQtqKeC98ivQDclOa121AbTl9pDip2W1H7TCcIWsbAgXD+BYnb8uQTSCYfgmAKMNwYswtKUzkPrQmCPA1c7/kADgY2WmtXGGPWZHFskTAB5dw/i0o1/hkFUwe9b9tRYXpHCuURGBWIv9xpJ/j97xIn6vr6eKXSDRvUoew7302s3LhhffP2U5A20VIhANBQAc+erv+7d4dvHwCR59BaJizjPDlovsF7XIkaBV+Pgvc/8V4fjjSJNOG17c0UVI7Cn8D/A13GyX2XQB/hPhKdx1HkL/hpYHuyoOiKSmE7ipac0zKttY3oan8RVRx52Fr7qTHmGmPMNd5uE9FyeT4qa/XNTMfmOqa2oQKVk5wDvIVi5Y4g0SdQjfwELeUDVLBmCKogVgIZORUVioWORDTRRyKp9YZiMVi9StnICxcogWzBglQ/QhgtFQLl5cpnGDRI2sjgIXD5FRA5BkUJ3U/LosRiKDhuFOr6stV7zEZG9SLhXlJbYt4est9c1EFtIlrt+81yTgDGknjJ+ia1GuLd0S7K98Ad+SQvgdjW2onoEgluuyPwvyWxFmPGYzsOf0JZPO+hG/9rwLUtPMdCFKfnh2Xch/oOJheR74Tsuy8M3kk1iaJRdXMKalhNTfDOO5r8y8qkCVRkUXAuEpEQSS7sBTpPz16qAb91q/ouH3YYHDCmmXNHUEW4fdGSORuaSC1ctB1pCkWC38cgqGSFBTzdgy5Rfz+LFOSJyOlcSWLOwVDUd7kGJe07R3FR4zKLc6IbCqJej+6edFW+MvECiZNFHSqCV8AEpfakV+94Y5oD9ocPP9RqPhKR+WjBgsR46kyx1aCJ/pBD4IMPUhvinHACHHZ4DiWtd0chMRegIjs7onTch8jcuaac+G9cTmLBngJzNbJ6+pN8NYkRQD5hH88XClciH4HvIC5Den/hGm45WogTBHkhl3jeKlKdh+myVzs5Y09W74LVq2CHPqpbtGRJ4j5+VJBv+jEmnnZfVhZvnTl9erwZuDFyvB1+RB4GOYB4pxifzch3lK5NWX/iTYVjSCOoI7wgUDuzO7JM/g5lGF+Kiu4lczHqkOYrrtWorSZo/fMhUmI3I3OR8wk0T6wJ3nwTFn4OvXoqYq6NnMHN4ZrXt4gNyGwzhPzJ0E2o9vFKlIXjL8mKyI5cKL5YDP/4R6IW0Lu3bpgZH8mUc8gh8PHHXv5AXzjpJOjaVU7l55+X03mXXaQNRNoqyWs2yqtM7l4PMvvNIzFdphrZU9IVBSpSpgK/QJfs15E2UYLrlbzx+GPxzn1lZSokd/31UNl2CwTXvD5nbgR+jUxANSjmbg/iXb6707qvswYtp/4P5fmPI7GZbQkzZGc48kh4/XXdKBUVcN75irHec8/4foNCSmf27JUaetdmjES/4WgSU3BByYbJs2Udif0kOwhjUEObbHkbWdFWItfK47gsY5/GBoVT+wvxWEymzLlzVXBx7Rp1/RszJv+l1kNwGkFWvIlCI4LxcbujesVfQcbRMpS/f3E7jqtE2LYNttVCD68ERdFyO/BdpNkFKUfXh6/ZVKMZ9aj2G1p7swLdIr6SVI4a/83FaREgQXDjjYmh0NEodOsmk2hjo56PHKl8mjzhNIKc+IhEb5lF6v7JqJyA/2Neg8JK92nX0XV6qqr0KHquRYH595DoL2gifv1EUQf7TiwEQH6H4EK2CVnHVlO6tRr9Bktr1siXteuuqsPV2BivqrtlS9z/1dAAn34qc2d11zYdmusumhW7kurQ7YciSII3fBmKJnGULj8mPHPYv04iwJHtN5xC0ZvUyNkYsoSWIrEmuPcedeOrr1eJk6VLVT9r8GDYay84eVxqLk1YP/A2wAmCrBgHnIVU+h7IH/AIqeUFDOGtnsJoQmEYl6CQjZYUPHMULyNQd5Z0+K05JpOmrFbn4HDgaJRVHEG3zq9JvWVKhQ0blbeSbIrfbZgSF8/+qlpYRiLx8ObycvnD8lxXKAxnGsoKg1IwvwesQeafvijb9EK0AoyhuLuwer4gnfh9JEiOQNU0JiK/QxUKP3yNcNk8HQVrr0B31wRKd2nVEdgVJQqGTfS16Lp5ALXheJxOaTQvQ4VlHkXW0wNRhe5SpUtlaoJjLKbtPpWVcMWVqp315TpluZ96ag55L9njnMU5Mx/F1Q1Edt+wH20qcQERQyEU00jUArqi5LQDko5dhqJS/Jr3lShTJzmW3VFYXkRZ4d2B05Ggr0e/dwP6fZMjirqhUp7j2m+YjsLx4gvxarjRKOyyK5x3XrtM9D7OWdxmDKP5ksMXoeBrnw9JXS2m6/r9StK+21GpgyJJSHKgzOIr0GrfAA8Cz6EggwjKPp6Pgu+Dv6VfyrwTswKV5xrqPUqZr5wEOw9VL+LevWGffdpVCGTCCYJ2YVnS8zpUqGUjWi2Wo5Vkcvdw0EoyGUNRdsAqOV5EJsPniIcWW7TyfxoVq/M5GJUnn0NiBNpBbT/MtmATiqqOImtlWD+mR1CmchQpR/8NfL+dxleMGAMjRuhRZDhncbuwH4mRJNXIYTgOZSkfh7qAV4cc6+/TJXDszyiamvYlyyMogOCfJGp7oIk+rOv7c6ivcQT9nncAfZBJ6WHCmwMXIYtRjsAFqF3zaBK/gqeAs1Fnslq03tkG/BzVaHQUHU4jaBceQgVYFqJJ4nvoLsom87ULCsr+C7oDj0cTkKOwBLuwJFOBfEDHodyCc7ztfkbVZqTpzUT+H19DGID8SUXSq2AeUmo2IqvWqd7264G1xMNDP0fKzm9RsbnvEv7VVKBbYLe2G7KjdThB0C7siBrdr0YOwpYmh3TF1R5qKyxaws5B3d1PJbsonuTsYYCeKGhgAfCut+099NtXo0izC4i3Mr2axKX0EpRs9usWfYI2YSGKW9iCvqJnUeL0xejjBXMEgv2Yfk16+ViPqrI4ig5nGmo3DEqpbNsMQUdLuRI583+ObBnXZ3ncNSSa8qpQOPAuJAqJbcB/IXPet4BDiUeLBXoog7c9qdpqofgbif0HaonLp8NJ7cfkJ0pnqhL+H8jK6Sg6nCBwlDDzkY1/K9Do/b2beMnoTPwQeT/3QkHyT6FJ/r2QfW3g/LNRcD3IdBSM/OqKcguKAD/yNXkbqD7iIcjUE0WWSr8k9eWEu7qgZQXrHO2KEwTtwgI04SSHgjoKy5ekRl9FUanx5jDI1zMT+XD8Cbw5J34d6mxnUd/r47xjosi4XiQd3i8gcUKvBq7y/u+Kiu8uR76C+4l/7P9BK/8wN8fqNhmpIw/kJAiMMb2NMS8ZY+Z5f1M6tBhjBhtjXjXGzDbGfGqM+U7gtV8ZY5YZYz7yHp0ws+Y5lEB2NapgWoNWkG8UclAOQKv5oCAwyMSTiyF7HFoqB8+ZzCfAJDSjPofMR3VIwyiOuHIOQEM7BDVU+xWynvkYFAGdnOBe5u13B6mWs054d3cWctUIbgBettYOB172nifTCPzAWjsSXVbXGWMCxeT5g7V2tPfooL2LM3ERMrBuRgbULSgy5BgUOngtrs5QoeiGynqMQEbvPVF2dy4FcW4lXn02QvgtAfLG+kQpSuX8GOTz/gT4ES2TUecBP0XCIIpabPw+4xEOUKe9WHK1vrYn16ih8ehyAWXWvAb8JLiDtXYFnlfMWrvZGDMbGATMyvG9C8TLwGMoQuRbKEokHY0o9i4Mi0wT96LV4N/zOEZH9uyN7Pb54ofAZ97/FcAMFC8ZDKC3KPi+k/Mz71Ei7bdzoqkRnngCZnnT4v77wymntEtTGsh9GdLfm+j9Cb9fpp2NMUNRdtX7gc3XG2M+NsbcHWZaKi4eRHVkbkfLm31R+6V0RFCceKaveRsSLI6Ozyok0P2aQrVIw/h60n6NlFRCoBMCzfPqqzBnjqqTWqv2q++FBR60Dc0KAmPMZGPMzJDH+Ja8kTGmG5rxvmut9YOnb0fLpdFIa/h/GY6/yhgz1Rgzdc2aNS156zxyA/EgaX+1f3czxzxLeD+DINmaIhpRotIUMsfpOfLLZhQWcxTwHcJ7E/v7JTufy1FCYZBG5A/o5NyD2lseiosYao6FCxP7DjQ0wIL2S8Nu1jRkrT0h3WvGmFXGmIHW2hXGmIGkiQswxkSREHjAWvt44NyrAvv8Dc2a6cYxAdVfZsyYMQUKvUkuCtdI82UB/GzSjWh1eD+6K+q946uBm7N4782oqMs87/lg4C3UAcTRdjSh+skzkS/nA1QO5D1ShftQlCuy2DvOrwkVlhswp22GWyz8HaVk+LfH2cj5XKqlqK2FNas1wffrr+qjQWpqYOXKeL+CsjK1Zm0ncjUNPY06q+D9fSp5B2OMAe4CZltrb0l6LWhgPxPdbUXMhSSGQnQhu0bzBvkUxhMv0P6faJX5NHBZFuf4JXKrbPEeC4AfZDvwAE+gaqmDkAew/R1THYtZyObvO/S3I59C0K/Q6G2PIGF/ONALWUHfJLwiW6bmNZ2AW0lcI21DkUSlSKwJHvgH3Hkn3Hcf3Ppn2LghcZ+TxkKXKqio0KNrVzi2/aRmrs7im4GHjTFXoCyccwCMMTsCd1prx6G74uvAJ8aYj7zjfupFCP3OGDMauZMWoRjLIuZ/0Vf2CIo4uYXU/gHZ0Af4RQuP+ZjE6KJ6Wi433yAexQRwG1oL/LaF5yklMimfFpkLb0HZV4ch39FkEk1E45B5yFf9K8lO+HdgwmaWipBtpcCUqfHexCCt4Kmn4OJL4vv06gXXXw8L5stBPHwYVLZfmfmcBIG19ktCWnJZa5fjRQ1ba98ijbvIWpvsRStyIkgY/G8B3nsM8DbxqpaVZCeEXkWBXFtRlk9wmVaLHOBOEKRnL1RqcxYSxJUoz2Ak8A+09PUn+LdQ97l+aO0zBpXnvA25wF4jnohWJIljyXyODLSVaOiZLI9rkX98V1LdXL8AziVuTa1GH7sUWb0q0f5vrXoWJ1NdDfvs237jCuCKznUI3kNhiJXIlBNFMe/NBWZPQ0XU/Mk/iiai4Co329pHc9Gdvzel1SazHE3gN6BGM/sDN3nbJ5PqI2pCk/7v0Hc7ERnMJ6NZMUrR3nbTkRuqESmKv0QfuX/Ivn9AGcQV6OO8QGJrhVOR1fOv3j4/ILzdRikwYKB8Ag1egIcx0C9jgGW7U6RXpCPOdKR0+RNOF7Tc+jHNu3geIHGiavCOKUMTVhXNCxOLCqzdj+7oMlQqowTi4P9NDVrVJ7MzEs7pEgK3olaUvyB82VxkfJvEgKh6pCzekrTfRyh7eDvxj34q0g6Cuv8J3qPUOeAAWLgA5s+XE7hrVxh/RqFHlYATBEXPX0mczOvQpJwuYxWk3/8UmZKS2Qn59bcg3f+wZt5/IhIo24jr+WfT+TuMLEEFd2aiyK8HkDkINPutQ7WB/omc/+nCeaPAhjSvFRnJMX+NhHfS/JTUgKkNqKJ2kbRSKCrKyuDcr8GG9dDQCDvsAOXFlUdShHntjkTCnJWZHJirkO/gYVLDFqtRVbD/Qsu85oQAKMwxeZLLpjpnIXkaNXmpAk6m5RNxI8oZeNc79iPgSDTTPYgiwHZFguF+1GFsJNLWgktiv3bRSDoEp5GotFSjQLdkhpEabFZF3GL4LComcyByozhkDurVWyahIhMC4ARBB+BqUstA/ijD/k8irSFYQ7gc1dp/BEUNtYSwwmy7tvAca9ByM9v0jxjSalY1t2MIH6C6yKvQ9zAZ+FoLz/E5GrM/2/llpJ8FvuGdtw59pvHIK/op8sk8CYxC/oFRKFKrCjnthyHv61dRXkiRcRP6qirR8H+GLptkDkZmpCqkAXRDUckGeMk7x/uopNbVSHY6ihonCIqeA1B9o1NQNs7dZA49DJtsI+hubE35x5PQ3VyJOmv1QXd9NtSjkhw7oY4kx6JomSFIG/k05Ji1aALdG9ngLyK1MH4mridxudqIfBotoTvxSKDgeZaRak3diMZskAP/dKRBbAE+RBFHc5ERfQGwHgmUc1s4pnagAvm169Dwf5ph35tQ/MJTqH7ecd7220gNTPtT3kfqyDNOEHQIDkGTxys0v7odjyZt/6cNFpJvKRuRjfyXaBJ7FbWu+oRw43EyN6IVeT2yq7+BtJIlKBLqCFIN01cic1Std8wTwJ0tGHNYbkVLXWEDgEuJR1RVIyHmh9QEKUPJY5mYTKKA3o6Wzh2gN8UqtBbohzSBYEL0cPSV9PWe1yEZmExy1Q1H0eGcxZ2OgagW0U9QQbzTaF2/40dRg9oIWmHfi0IiZxO3g7+CYuXT8RaJZTks8cnPeud9jcTV8TQSfRK1SGhkK8xqSC0Fck7YjmjmehCt6I9FRm2f29Ey11/VX4xMbFegRPmIN857aX6mqyF1zVVF0Vdji6GvZR6Sf2tReugCwp3C3yS1+2YlMjE5ihonCDo076F+BmuAr6Dkpmpki86loukaZJIJhkWejy6XusC2Swg37/jsgUoshDV690kOqRyGtI1Y4PW9mh/yv7kFaRXb0MTdG/gjMqndiibtX6JJ/mDUrrLe23438UQvgwRUsgnnT+hzf4FMWNn4S85GTvpF6DutQkK1yFmChuwrQRYN/zfICnYKiTkGT5AaSfs11I/JUdQYazuAeprEmDFj7NSpUws9jAKzEJXB9ksed0ERMo+n3KCRYgAAF/JJREFUPSJ7JqG7N3htJCeigUwimdo6rkeT7RISBYjPCLTiDqbSf47KVW5DwmAU8pGE1etJxxvIlNYL+TeeRL0jfON1FfJ23kr8+8vm8+TCVmTiWoXyQlIS8ouPtagkVbIc74IUnEqkfO7mbd8JuVF8KlHD+55Ith+O1iuOgmGMmWatTVHjnUbQYXmRRKdoHfAM+ekC8jmpk75FM4A/oUeR7yITvVCNpP9GK/Wgycag8Mzkeiq7oFX6FKTdjCG72v1rkf+hHpnDgivuP5NaAe1FUu39bRnJ0xWVsO5A9EGukgeQHCtHl5x/CdShjOEnved/QMrSNuR47o3y6XyXTzVqeZkp6M1REJwg6LBUkzpBVpAfu/MOpGbMlqHuW37J7NEohr45uqA7/25kU2/0xjkOLRXD6EbL6hWv8MazGWkRP0dVQP2aBmExEWVJ2yuQ59ORwB0oheIDpCgGncUxEmMGzgF2RMpYb6RNXE1cBtcif8H3Kam+PB0BFzXUYTkbhXL4JR2rkR6eD05Cy0F/nVCFlnq/QUvDL9HMsEOW5+uBSmUMRoIqhlbk/5Wn8d6ETDrbkPDagrJ+fQ4OOWYxygruh4TVsWj56kjAIHfRn1ClkeSUluSI5MPRz/EjwqN+Y7gW3UWI0wg6LN3Q5PpntCIei2LY80F3FL1zKYr8iaIYQX8135p6wr9CJie88zQiB+pVKFwzF1aSauYJdrEbjNY8wZmpDoXatqjRXmnzbfQT3o4shech5SsdRyU9jyIlrTpk386OjcG776odZffucPwJKj1dJDiNoEPTExU0u438CQGfpcjpugU5fW9FIamtYQ3hLT0jtC57OJkzSF2qBr+Pr5DobK7EhbK0gjIUgOUXm/OjaNMxBFUlHYYiaI9DXcpKkRdfhNdegy++UIP6v02ALelanrY/ThA40vAoic7dWlpfOGY94VpEGcpKypXz0dK0GzI+9/L++l7NA1C+wI5I2zkdxf87WoUh+5njcJSHsBEJhWytiZ0Ja2Hq1HgZamv1/5zPCjuuAE4QONLQlVSPXms7Jg1FS8LkgmxPkR87gUFG6eFoiboM+D8U6O5HP53hbd+ECvJl24ehk9HxosU7J8YU1W/hBIEjhCaUZVtNYqmKG1t5vgqUQewXsBuKnM3HpT+kxXyIlp6+J3IbCk/9PO0RJcVdSBmKAieSuSDrW6gV93haXqbJkYoxsP8B8Yb1xqgC6Yg9CjuuAKXpLN6EaqL0BPah6DP925dtaIL+BH0x3VFc/sVoBmktu3vnbCuaSP0hDan1kkuQN5Cjtzbw/CIU5pnMm8h94u/7EsoYPimwTwy5fKYi2X4Nrp5Qc5w8FnrUxJ3FJ5wA3boXelT/JidBYIzxU0aGomT0c62160P2W4SCvJuARj+zLdvj88pM4rXDGlH4279wutG/uQlJyWDi2DZyEwLtwX4o+mg7SiqrRHUQdst0UGnwKonunnqUZhHG70nNvbuZREFwIWr5UIsii59B9n93D6XHlMHhR+hRhOT6090AvGytHY7qAGRqm3WstXZ0UnpzS47PD+ch3+UmdCE/jwsfT+ATEstBNJC5nlCxUAm8g4rb7Ic0mJdxsxOqB5Ts3knXlD5MgQpG5i5BmcS+sNiGvvaPcxmgo9DkepeMJx5+cS/yyLXn8S1nEYlOmloSsyVLngNJLARXgaJuOgJ9ULew6cAE4i2zSpxLkGLUFf201aia+GqUHP4Q8eoa3yY1Evf7ged+qYkg5SRqEY4OR64+gv7W2hUA1toVxph+afazwCRjjAX+aq2d0MLjMcZchVeLeMiQIa0f8Ug0T/i5RdWorpnD44fIiPwGsrHvhpLWHB2WKlS66XHkJD4W3fkjkPLX4D3/LarN9zAyB8VQLaEzA+cahiqdL0KaQjnxZmyODkuzgsAYM5nw1M+WVBk/3Fq73JvoXzLGfGatfaMFx+MJjwmg6qMtOTaBh1HG4wZ0A1xKuB6yEd08XYGDKKHaKBXIXrYIGZOHUUIfvvPSBbgg8Nxv5ezfSY1o0l+LKn+ckuY8EbRGuAyZg3ZHXc1KNBq3s9CsILDWnpDuNWPMKmPMQG81P5DUdlP+OZZ7f1cbY55AU+sbQFbH55VdUGONhagEzsCQfeaiRJh6tCraH0VPtKayQofEoC/K0WlZSmoceyPwv6hkVaZIuoHIOQxyKS0g3sXU0SHJ1UfwNLJA4v19KnkHY0xXY0x3/3+U7z8z2+PbhAqkFocJAX8kXyKH8hakGUxIs6/D0RE5kfCQz+SSTUHWoj7Fm5EQuQJVIv8q6s/zdp7H6Gg3chUENwMnGmPmoUvrZgBjzI7GmInePv2Bt4wxM1AW0XPW2hcyHV9wksvxb0MdGh2OzsJNpEYEVyHzUZg2cAeq3XcUKi/9exRtV4vMqJuRQHB0SFyHsjDGIVOQvzqqRjfC19vuLR2OgjAd+QZWI7/A/5CqKcxDzuBgLkIlEhjBSGOD7hkXsVu0uA5lLeEeFFnxOYqrvgBlYjocnY39UcJZJj4jnlcYJLl01K44IdBBcYIgjH7ICbYUaQPOCeYoZYahCLsgEeDHSIOoQFFDz7TzuBx5w8nvdJSheupOCDhKnZEokqgLirSrRmHY/4l6Ik0HvvD2c3RInEbgcDia50eoescSlDvQ19vem/TlKhwdBicIHA5HdgzxHo5OhzMNORwdhRdRQdWdgO+Rard3OFqJ0wgcjo7ANOAs4sXdJqBQTVcGypEHnEbgcBQba9Hq/wPiiY1PktpC+p/tPC5Hp8VpBA5HMTENNYjzk7NOQFVDu6G7NWgOqko52uFoFU4jcDiKifNRjauNqPb/ZBSqeRnQi/jSrZpiKcji6AQ4jcDhaA9qgQdRd7wTUBO1MJYmPa9DlXLPQ2Wf/wKsQ/6C49pkpI4SxAkCh6OtqUVN3r5App1fAQ8Q3gdjLxIbJ3UhLjT6o14BDkeecaYhh6OtuR9YjARCg/f32jT7Popi9atR6YZvo3pA76AsXoejDXCCIFdWAL9ApSiqkLq+pqAjchQb61CTo+Rt00P23RmYD8wCVgL7oB5BJ6Ouof8HHI2utZ2Al9tmyI7SwgmCXJgGDAf+GzWyqQPeBE4r5KAcRcfxpHa3awKORH2CkylHAsFv/rINOZC3oUby76BrbRlwOuoQ5nDkgBMEuXApiuwI0og6miWvAB3ZY9FqeBWp7RQ7Igeh0ua9AtuakInol6Rv0PoFqb0BLIldxMqAt/IySkcJ4wRBLixLs72CxBv4C2QTPgeFAjrSUweMBYaiVfFYEpufdFTORWWaeyRtr0ACL4ydydw60scVfXPkiBMEuXAoqSu2CuAW4k07VqCoj78hR+BlwM/QBLcXcgZ2hokuX/wCmde2e4830aq5MzASaQJBylEm8V2krux7oeiiaqAG+QUu855HUA+AUch/4HDkQE6CwBjT2xjzkjFmnve3V8g+exhjPgo8Nhljvuu99itjzLLAa+NyGU+7cx8wGt2UZchRPJnEiJAHgC3EJ4Ba1C/2JeQQvBPFiDvEOySWUthG52mK3ht4FuiJrpk+qM/vqcB30OLghqRjzkCa5x9R9NAnwFXAb4DbUHcxFwTuyJFcNYIbgJettcNR/ELyZYy1do61drS1djSKpq4Fngjs8gf/dWvtxOTji5odUD2YL9GE9TJyAAZpIHUVaInHiW9Dk8P2LN9zHXAxEkCXogSlzsQeJGpZUWBEgcbSFhyNfsMvgY+Af6A7Yqv3+CNqkRpkEXAdEohTUcG5CLoOkjVSh6MV5CoIxgP3ev/fS3iKTJDjgQXW2sU5vm9xUUNqVIjPWSgpyKeS1JvXkN0v0YAEzb+AGajo2FFkZ0fuKPwOGAx09x5DCI+saSlNSIP7BfAYhXVCG3TNrCH1uqlAjvIg9xGvOor3/61tNjpHCZKrUtnfWrsCwFq7whjTr5n9zyO1ZuL1xpiL0VrnB9bazrXG3QN4BfgBWgmejkoNrEATezVwOdmt7D5Fjmc/IqkerR5no3jz9sKPXPkEeA5Napcgk0eu9EGf8x00YR5KoiBtDRYJ5JfRqrsr8BqFL+E8nPAFwJ5Jz8vRd2GTtjkcecJYm3lpZIyZDAwIeelnwL3W2p6Bfddba1P8BN5rFcByYC9r7SpvW3/kKrPI6jnQWnt5muOvQtZRhgwZcsDixR1YqfgSfdrFwFeAa4g7lzPxCZoYgyGr1UiEtle/2P+HfvlgFcwKNIF/TGKIZC74IaQVyASXLS8jraIJ+C6yv3+INKng91aJvv/++RhsC/DDYochgTQVLQ5Woe/wSfQbB5mLjKpb0fdSDfwJ5Rg4HC3AGDPNWjsm5QVrbasfwBw0eQMMBOZk2Hc8MCnD60OBmdm87wEHHGBLkiZr7aHW2i5W30SVtfYIb3t78IK1ttqG/yoV1trfteKcz1prB1tra6y1Z1trN1lrN1prD7H6nBXW2gustY1ZnOsVq+/EH1O1tfZpa+2r3vmD4+1qrZ3bivHmwm+stZXeWHpaaz8IvFbXzLEzrb6HU621j7bJ6BwlADDVhtzBuZqGnkZGgZu9v09l2Pd8ksxCxpiB1jMtAWcCM3McT+emDK14f4NWuQeg1XnQvNCEbM+9Se+3aC2vkmirDlKPSiO0hBkovt4/57PARcjE9CFxB/qTqOrmt5s53x9Ibd7yO++8EeLmlXK0+t6lhePNhQ9QtJgfFgvSVvwcgspmjt8LRaA5HG1Ars7im4ETjTHzgBO95xhjdjTG/DsCyBhT7b3+eNLxvzPGfGKM+Rg4FnVidWSiCrgReB6Vtgg2J5mOjHi7oMn0sTy/9yAyN0N5lPRZsmFMJtHRvR2YBLxLYhRVLdllz6Yzr/UA3kB+lBrgYO95e4ZdziJ1fF+SmpnucBSAnG4Fa+2XKBIoeftyYFzgeS0hll5r7ddzeX9HgEbgJORx8bkYGIMyVPPBlSjvYSFKgkuOVlqHhIUv8ptz8vZAV2CwHEdXVFxtIfGw20rkdG+O76H8DF8rqAZ+7P2/F9JA2putqLzER6R+Xz3RGB2OAuMyizsLK0g120SQAzdfVCETx/WkX303ooicn2VxvvOJaxllaFL8P+AOZLqpQS0a9yAkQyWEY5AZ6CTU/OVhClsAsBYlgf0I/n97dx8rxVXGcfz7S3kJFwiVIhVqXzRFE9pIJKRSJBVt1XIVsWoj9a3GJqRNmthETdAmprExUaPGaNSIiMXGtJpYtFZqrG0jiQ3IS4DSUKCYoggC1kKtEmnl8Y/nbO+wd19m2ZnZe+8+n2Rzd+flzsPMcM+eM+c8h7V4wTYeLwCn4k1eeToJhFCyGJM4VsxgaJBazct4P/wiTcRHur7UYptTeNNLO5Px5qx78GaSdwKL0rr9eBfSCcBi8g+cegcjZ+au+/AZx2o1lDN4LeA3eC+vonpYhdClKAjGikl4PqOV+B/Nl/FUF/NKONZ0/IFr/YjpmnF4807NKbxZZjJwJWd/C56C1zDqTcW/2Y9mJxheYJ5iqLALYYSIpqGx5GN4U9CP8WRtXy/pOJ/Bm21qPXEm4Q+pa6OBZ+KJ98DTI1yO/1FfiD85GksjoVu5jrNrMhPxWk8II0zUCMaay9OrTBfjHX3vwR8afxAfJfsHvPnjGrxAAM+HdJSh2sNGvObSbKrGsWQeng7kNuAkXjCsa7lHCD0RBUE4N7OBL9Qta5QOeS9nNyH9Bx8h3coZPJne+Yz+VArvTa8QRrBoGgrlupKz/5gP4D1pmtmCp32YjRcEvy0vtBCCi4IglGsdPo5hCv4s4T14kr1G/svQWIjT+DwOH2J4Ns4QQqGiIAjlmg08DWzCR9f+jOZ33UGG97IZh2cjzdqE1zRejRcUJ4sKNoT+FM8IQvnG4yN727mQ4QXBaXzQWc1BvOfNi+nzr/EsVY91GWMIfSxqBGHkmIYnjpuE9zoawKdwzM5Q9jhn5+U/jfdWyqapCCF0JGoEYWS5De9+uhsflFafOX0Kw9MyjCPu5BC6EDWC0N4hvAvkG/BBaydKPt4VwIcZXgiA5w66jKGEdgPAl4g7OYQuxPeo0Nq/8bTNtUFhB/GHv1voTcK0icBm4Ad4zqMlRD/9ELoUBUFo7U/4g9naoLDTeO+fQ/gI414YIGauCKFAUaEOrU1geFbTMxQ/+1kIoWeiIAitvQV/NpBtk19G9ZO+hxBKE01DobVxeCbTrwB7gKvxLp0hhDEjCoLQXq1nTghhTOqqaUjSjZKeknRGUqPOfrXtrpe0V9IzklZllk+X9Iik/elnzNkUQggV6/YZwW7gA7SYmFDSecB38STFc4GbJM1Nq1cBj5rZHOBR8s1MG0IIoUBdFQRmtsfM9rbZ7CrgGTP7s5mdBu4Hlqd1yxmaqmMd8P5u4gkhhNC5KnoNXQT8NfP5EENpxC40syMA6efMZr9E0kpJWyVtPX78eGnBhhBCv2n7sFjS7/EZaevdaWa/ynGMRuNPrcGylsxsNbAaYMGCBR3vH0IIobG2BYGZXdflMerHoL4WOJzeH5U0y8yOSJoFHOvyWCGEEDpURffRLcAcSa/Ds8OsAD6S1j0I3Iz3Ur8ZyFPDYNu2bf+QdLCEWJuZgc+bNdJFnMWKOIszGmKEsR/npY0WyuzcW1kk3QB8B58r6gSww8zeLWk2sMbMBtN2g8C38Nlr15rZl9PyC4CfA5cAfwFuNLN/nnNAJZG01cyado8dKSLOYkWcxRkNMUL/xtlVjcDM1gPrGyw/DAxmPm8ANjTY7jng2m5iCCGE0J3INRRCCH0uCoJ8Vvc6gJwizmJFnMUZDTFCn8bZ1TOCEEIIo1/UCEIIoc9FQRBCCH0uCoIkTyZUSW+UtCPzekHSHWndXZL+llk3OPwo1cSZtntW0pMplq2d7l92jJIulvS4pD0pg+2nM+tKPZfNsuFm1kvSt9P6XZLm59234jg/muLbJekJSfMy6xpe/x7FuUTSycz1/GLefSuO83OZGHdL+p+k6WldJedT0lpJxyTtbrK+nHvTzOLlz0m+BqxK71cBX22z/XnA34FL0+e7gM+OlDiBZ4EZ3f47y4oRmAXMT++nAvuAuWWfy3TdDgCvxyfc3Fk7bmabQeBhPD3KQmBz3n0rjnMR8Kr0fmktzlbXv0dxLgEeOpd9q4yzbvtlwGM9OJ/XAPOB3U3Wl3JvRo1gSKeZUK8FDphZlSOcofuMrVVkfG17DDM7Ymbb0/t/4fOfXVS/XQlaZcOtWQ78xNwm4Hx5CpQ8+1YWp5k9YWbPp4+b8PQtVevmnIyo81nnJuC+kmJpysw2Aq0G1ZZyb0ZBMCR3JtRkBcNvlNtTdW1tGU0uSd44DfidpG2SVp7D/lXECICky4A3A5szi8s6l62y4bbbJs++Ren0WLfg3xRrml3/ouWN82pJOyU9LOmKDvctQu5jSRoArgd+kVlc1flsp5R7s6+mqlSLTKod/p4JwPuAz2cWfx+4G79h7ga+AXyqh3G+1cwOS5oJPCLp6fRtoxAFnssp+H+4O8zshbS4sHPZ6JANltX3oW62TSGZdHPKfSxJb8cLgsWZxaVe/w7j3I43ob6Ynvf8EpiTc9+idHKsZcAf7ex0N1Wdz3ZKuTf7qiCwFplUJXWSCXUpsN3MjmZ+9yvvJf0QeKiXcZqn+cDMjklaj1cdN1JQxtciYpQ0Hi8EfmpmD2R+d2HnsoFW2XDbbTMhx75FyRMnkt4ErAGWmqdsAVpe/8rjzBTwmNkGSd+TNCPPvlXGmTGstl/h+WynlHszmoaG1DKhQvtMqMPaD9MfvJob8Gk8y9A2TkmTJU2tvQfelYmnk39nmTEK+BGwx8y+WbeuzHP5SjbcVLNbkeLNehD4ROqhsRA4mZq48uxbWZySLgEeAD5uZvsyy1td/17E+Zp0vZF0Ff5357k8+1YZZ4pvGvA2MvdsxeeznXLuzbKfgo+WF3ABPm/y/vRzelo+G9iQ2W4Av4mn1e1/L/AksCtdgFm9ihPvObAzvZ7CJxFquX8PYlyMV113ATvSa7CKc4n3vNiH97K4My27Fbg1vRc+z/aBFMeCVvuWeE+2i3MN8Hzm/G1td/17FOftKY6d+EPtRSPxfKbPnwTur9uvsvOJf8E8AryEf/u/pYp7M1JMhBBCn4umoRBC6HNREIQQQp+LgiCEEPpcFAQhhNDnoiAIIYQ+FwVBCCH0uSgIQgihz/0fXUSSE4yfBWYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "np.random.seed(100)\n",
    "\n",
    "def gen_spiral_dataset(N=100,D=2,K=3):\n",
    "    N = 100 # number of points per class\n",
    "    D = 2 # dimensionality\n",
    "    K = 3 # number of classes\n",
    "    X = np.zeros((N*K,D)) # data matrix (each row = single example)\n",
    "    y = np.zeros(N*K, dtype='uint8') # class labels\n",
    "    for j in range(K):\n",
    "        ix = range(N*j,N*(j+1))\n",
    "        r = np.linspace(0.0,1,N) # radius\n",
    "        t = np.linspace(j*4,(j+1)*4,N) + np.random.randn(N)*0.2  # theta\n",
    "        X[ix] = np.c_[r*np.sin(t), r*np.cos(t)]\n",
    "        y[ix] = j\n",
    "    return X,y\n",
    "\n",
    "N = 100 # number of points per class\n",
    "D = 2 # dimensionality\n",
    "K = 3 # number of classes\n",
    "\n",
    "X_spiral,y_spiral = gen_spiral_dataset()\n",
    "# lets visualize the data:\n",
    "plt.scatter(X_spiral[:, 0], X_spiral[:, 1], c=y_spiral, s=20, cmap=plt.cm.spring) #s=40, cmap=plt.cm.Spectral)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6.2 softmax函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def softmax(x):\n",
    "    e_x = np.exp(x)\n",
    "    return e_x / e_x.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.8360188 , 0.11314284, 0.05083836])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = [3.0, 1.0, 0.2]\n",
    "softmax(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-2-e3aa77d695fd>:4: RuntimeWarning: overflow encountered in exp\n",
      "  e_x = np.exp(x)\n",
      "<ipython-input-2-e3aa77d695fd>:5: RuntimeWarning: invalid value encountered in true_divide\n",
      "  return e_x / e_x.sum()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([ 0., nan])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = [100,1000]\n",
    "softmax(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因此，可以先求出所有${z_i}$的最大值a，然后用${z_i-a}$去计算softmax()函数值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0. 1.]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([7.12457641e-218, 1.00000000e+000])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def softmax(x):   \n",
    "    e_x = np.exp(x - np.max(x))\n",
    "    return e_x / e_x.sum()\n",
    "\n",
    "print(softmax(z))\n",
    "z = [500,1000]\n",
    "softmax(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.00548473, 0.01490905, 0.04052699],\n",
       "       [0.8140064 , 0.01490905, 0.11016379]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = np.array([[1, 2, 3],[6, 2, 4]])\n",
    "softmax(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对多个样本，应计算它们各自的softmax()函数值，即对每个样本单独计算其softmax值向量。正确代码应该是:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.09003057, 0.24472847, 0.66524096],\n",
       "       [0.86681333, 0.01587624, 0.11731043]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def softmax(x):   \n",
    "    a= np.max(x,axis=-1,keepdims=True)  \n",
    "    e_x = np.exp(x - a)\n",
    "    return e_x /np.sum(e_x,axis=-1,keepdims=True)\n",
    "\n",
    "softmax(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.09003057, 0.24472847, 0.66524096],\n",
       "       [0.86681333, 0.01587624, 0.11731043]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def softmax(x):\n",
    "    e_x=np.exp(x-np.max(x,axis=-1,keepdims=True))\n",
    "    return e_x /np.sum(e_x,axis=-1,keepdims=True)\n",
    "\n",
    "softmax(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般的，假设$\\pmb z = (z_1,z_2,\\cdots,z_k,\\cdots,z_C)$，用$f(\\pmb z)$表示$softmax(\\pmb z)$函数，则有：\n",
    "\n",
    "$$f_i = \\frac{e^{z_i}}{\\sum_{k=1}^C e^{z_k}}$$\n",
    "\n",
    "其中，$\\sum_{i=1}^C f_i = 1$。\n",
    "\n",
    "$\\pmb f$关于$\\pmb z$的梯度可如下计算："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax_gradient(z):    \n",
    "    f = softmax(z)\n",
    "    grad = -np.outer(f, f) + np.diag(f.flatten())\n",
    "    return grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用df表示某个其他变量$L$关于$f$的梯度，则$L$关于$z$的梯度的python计算代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax_backward(z,df):    \n",
    "    f = softmax(z)\n",
    "    grad = -np.outer(f, f) + np.diag(f.flatten())\n",
    "    \n",
    "    return df@grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6.3 softmax回归\n",
    "\n",
    "$$f(\\pmb x) =  softmax (\\pmb x\\pmb W_{,1},\\pmb x\\pmb W_{,2},\\pmb x\\pmb W_{,3}) =  softmax(\\pmb x \\pmb W) $$\n",
    "\n",
    "$$L(\\pmb W) = -\\frac{1}{m} \\sum_{i=1}^{m} log({f^{(i)}_{y^{(i)}}}) $$\n",
    "\n",
    "$$L(\\pmb W) = -\\frac{1}{m} \\sum_{i=1}^{m} log({f^{(i)}_{y^{(i)}}})  = -\\frac{1}{m} sum (log \\pmb F_{\\pmb y}) $$\n",
    "\n",
    "计算交叉熵的python代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cross_entropy(F,y):\n",
    "    m = len(F) #y.shape[0]\n",
    "    log_Fy = -np.log(F[range(m),y])    \n",
    "    return np.sum(log_Fy) / m  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用这个函数计算例子中的交叉熵："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8573992140459634\n",
      "0.8573992140459634\n"
     ]
    }
   ],
   "source": [
    "F = np.array([[0.2,0.5,0.3],[0.2,0.6,0.2]])  #每行对应一个样本\n",
    "Y = np.array([2,1])\n",
    "\n",
    "print(-1/2*(np.log(0.3)+np.log(0.6)))\n",
    "print(cross_entropy(F,Y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于一个样本，如果其对应的$y^{(i)}$的第j个分量$y^{(i)}_j = 1$，即该样本属于第j类，则该样本的交叉熵损失可写成：\n",
    "\n",
    "$$-\\log(f^{(i)}_j) =-y^{(i)}_j \\log(f^{(i)}_j) = - \\sum_{j=1}^C y^{(i)}_j \\log(f^{(i)}_j) = -y^{(i)} \\cdot log(f^{(i)}) $$\n",
    "\n",
    "针对one-hot形式表示的目标值，可以将所有样本的交叉熵损失写成如下形式：\n",
    "\n",
    "$$\\begin{aligned}L(\\pmb W) &= -\\frac{1}{m} \\sum_{i=1}^m y^{(i)}\\cdot log(f^{(i)})= -\\frac{1}{m} np.sum(Y \\odot log(F) )\\end{aligned}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对应的python计算代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8573992140459634\n"
     ]
    }
   ],
   "source": [
    "def cross_entropy_one_hot(F,Y):\n",
    "    m = len(F) \n",
    "    return -np.sum(Y*np.log(F))/m  # -(1./m) *np.sum(np.multiply(y, np.log(f)))\n",
    "\n",
    "F = np.array([[0.2,0.5,0.3],[0.2,0.6,0.2]])  #每行对应一个样本\n",
    "Y = np.array([[0,0,1],[0,1,0]])\n",
    "\n",
    "print(cross_entropy_one_hot(F,Y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6.5 通过加权和计算交叉熵损失"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(Z):\n",
    "    A = np.exp(Z-np.max(Z,axis=1,keepdims=True))\n",
    "    return A/np.sum(A,axis=1,keepdims=True)\n",
    "\n",
    "def softmax_cross_entropy(Z,y):\n",
    "    m = len(Z)\n",
    "    F = softmax(Z)\n",
    "    log_Fy = -np.log(F[range(m),y])\n",
    "    return  np.sum(log_Fy) / m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "31.500003072148047"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Z = np.array([[2,25,13],[54,3,11]])  #每行对应一个样本\n",
    "y = np.array([2,1])\n",
    "softmax_cross_entropy(Z,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果目标标签是one-hot向量形式，则下面代码从加权和计算交叉熵损失："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax_cross_entropy_one_hot(Z, y):    \n",
    "    F = softmax(Z)\n",
    "    loss =  -np.sum(y*np.log(F),axis=1)\n",
    "    return np.mean(loss)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "31.500003072148047"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Z = np.array([[2,25,13],[54,3,11]])  #每行对应一个样本\n",
    "y = np.array([[0, 0, 1],[0, 1, 0]])\n",
    "softmax_cross_entropy_one_hot(Z,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6.6 softmax回归的梯度计算\n",
    "\n",
    "$$\\nabla_Z{\\mathcal L} = \\pmb F -\\pmb I_i$$\n",
    "\n",
    "或者\n",
    "\n",
    "$$\\nabla_Z{\\mathcal L} = \\pmb F -\\pmb Y$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def grad_softmax_crossentropy(Z,y):\n",
    "    F = softmax(Z)\n",
    "    I_i = np.zeros_like(Z)\n",
    "    I_i[np.arange(len(Z)),y] = 1   \n",
    "    return (F - I_i) / Z.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def grad_softmax_cross_entropy(Z,y):\n",
    "    m = len(Z)\n",
    "    F = softmax(Z) \n",
    "    F[range(m),y] -= 1\n",
    "    return F/m  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5.13090829e-11,  4.99996928e-01, -4.99996928e-01],\n",
       "       [ 5.00000000e-01, -5.00000000e-01,  1.05756552e-19]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Z = np.array([[2,25,13],[54,3,11]])  #每行对应一个样本\n",
    "y = np.array([2,1])\n",
    "grad_softmax_cross_entropy(Z,y)\n",
    "#grad_softmax_crossentropy(Z,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了确保分析梯度计算没有错误，可以用1.4)节的通用数值梯度函数计算交叉熵关于Z的数值梯度和上述的分析梯度进行比较："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_grad [array([[ 0.        ,  0.49999693, -0.49999693],\n",
      "       [ 0.5       , -0.5       ,  0.        ]])]\n"
     ]
    }
   ],
   "source": [
    "def loss_f():\n",
    "    return softmax_cross_entropy(Z,y)\n",
    "\n",
    "import util\n",
    "Z = Z.astype(float)        #注意：必须将整数数组转化成float类型\n",
    "print(\"num_grad\",util.numerical_gradient(loss_f,[Z]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果样本目标是使用one-hot向量表示的，则计算交叉熵关于Z的梯度代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5.13090829e-11,  4.99996928e-01, -4.99996928e-01],\n",
       "       [ 5.00000000e-01, -5.00000000e-01,  1.05756552e-19]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def grad_softmax_crossentropy_one_hot(Z, y):   #y是用one-hot向量表示的\n",
    "    F = softmax(Z) \n",
    "    return (F - y)/Z.shape[0]\n",
    "\n",
    "Z = np.array([[2,25,13],[54,3,11]])  #每行对应一个样本\n",
    "y = np.array([[0, 0, 1],[0, 1, 0]])\n",
    "grad_softmax_crossentropy_one_hot(Z,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "则损失函数关于权重$\\pmb W$的梯度的向量形式是：\n",
    "\n",
    "$$ \\frac {\\partial{\\mathcal L}} {\\partial{\\pmb W } } = {\\pmb X}^T {(\\pmb F-\\pmb Y)} \\tag{3-61}$$\n",
    "\n",
    "同样，可以给softmax回归的交叉熵损失添加正则项，如果用整数表示目标值，即损失函数变为：\n",
    "\n",
    "$$L(\\pmb W) = -\\frac{1}{m} \\sum_{i=1}^{m} log({f^{(i)}_{y^{(i)}}}) +\\lambda \\|\\pmb W\\|^2  \\tag{3-62}$$\n",
    "\n",
    "如果用one-hot向量表示目标值，则即损失函数变为：\n",
    "\n",
    "$$L(\\pmb W) =-\\frac{1}{m} \\sum_{i=1}^m y^{(i)}\\dot log(f^{(i)}) +\\lambda \\|W\\|^2  \\tag{3-63} $$\n",
    "\n",
    "\n",
    "则损失函数关于权重$\\pmb W$的梯度是：\n",
    "\n",
    "$$ \\frac {\\partial{\\mathcal L}} {\\partial{\\pmb W } } = {\\pmb X}^T {(\\pmb F-\\pmb Y)} +2 \\lambda \\pmb W  \\tag{3-64}$$\n",
    "\n",
    "根据梯度的计算公式，可以很容易写出损失函数关于$\\pmb W$的梯度的计算代码。下列代码中，假设X表示多个样本的数据特征矩阵，y表示目标值向量，reg是正则化参数$\\lambda$，loss_softmax()和gradient_softmax()分别计算损失函数的损失和关于$\\pmb W$的梯度："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_softmax(W,X,y,reg):\n",
    "    m = len(X)\n",
    "    Z=  np.dot(X,W)\n",
    "    \n",
    "    I_i = np.zeros_like(Z)\n",
    "    I_i[np.arange(len(Z)),y] = 1\n",
    "    F = softmax(Z)\n",
    "    #F = np.exp(Z) / np.exp(Z).sum(axis=-1,keepdims=True)\n",
    "    grad =  (1 / m) * np.dot(X.T,F - I_i)      # Z.shape[0]\n",
    "    grad = grad +2*reg*W\n",
    "    return grad\n",
    "\n",
    "def loss_softmax(W,X,y,reg):\n",
    "    m = len(X)\n",
    "    Z=  np.dot(X,W)   \n",
    "    loss =  softmax_cross_entropy(Z,y)+reg*np.sum(W*W)\n",
    "    return loss  \n",
    "\n",
    "\n",
    "def loss_softmax_(W,X,y,reg):\n",
    "    m = len(X)\n",
    "    Z=  np.dot(X,W) \n",
    "    Z_i_y_i = Z[np.arange(len(Z)),y]\n",
    "    negtive_log_prob = - Z_i_y_i + np.log(np.sum(np.exp(Z),axis=-1))\n",
    "    loss =  np.mean(negtive_log_prob)+reg*np.sum(W*W)\n",
    "    return loss  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用下列数据测试一下这个函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.30213245 -1.75779321  1.69566076]\n",
      " [ 0.5254108  -2.19194012  2.22652932]]\n",
      "2.0863049636282662\n",
      "2.0863049636282662\n"
     ]
    }
   ],
   "source": [
    "X = np.array([[2,3],[4,5]])  #每行对应一个样本,每个样本是2个特征\n",
    "y = np.array([2,1])             #类别数是3\n",
    "W = np.array([[0.1,0.2,0.3],[0.4,0.2,0.8]])  #  2X3矩阵\n",
    "\n",
    "reg = 0.2;\n",
    "\n",
    "print(gradient_softmax(W,X,y,reg))\n",
    "print(loss_softmax(W,X,y,reg))\n",
    "print(loss_softmax_(W,X,y,reg))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果用one-hot向量表示每个样本的目标值，y表示多个样本的目标值构成的矩阵，则下面的loss_softmax_onthot()和gradient_softmax_onthot()分别计算损失函数的损失和关于$\\pmb W$的梯度："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_softmax_onehot(W,X,y,reg):\n",
    "    m = len(X)   #样本数目\n",
    "    nC = W.shape[1]   #类别数目\n",
    "    #y_one_hot = np.eye(nC)[y[:,0]]\n",
    "    y_one_hot = y\n",
    "    \n",
    "    Z = np.dot(X,W)      # Z是加权和\n",
    "    F = softmax(Z)       # F是概率矩阵\n",
    "    grad = (1 / m) * np.dot(X.T,(F - y_one_hot)) + 2*reg*W \n",
    "    return grad\n",
    "\n",
    "def loss_softmax_onehot(W,X,y,reg):\n",
    "    m = len(X)  #First we get the number of training examples\n",
    "    nC = W.shape[1]\n",
    "    #y_one_hot = np.eye(nC)[y[:,0]]\n",
    "    y_one_hot = y\n",
    "    \n",
    "   # y_mat = oneHotIt(y)  #将整数类编码转换为one-hot向量形式\n",
    "    Z = np.dot(X,W)       # Z是加权和\n",
    "    F = softmax(Z)        # F是概率矩阵\n",
    "    loss = (-1 / m) * np.sum(y_one_hot * np.log(F)) + (reg)*np.sum(W*W) \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.30213245 -1.75779321  1.69566076]\n",
      " [ 0.5254108  -2.19194012  2.22652932]]\n",
      "2.0863049636282662\n"
     ]
    }
   ],
   "source": [
    "X = np.array([[2,3],[4,5]])  #每行对应一个样本,每个样本是2个特征\n",
    "y = np.array([[0,0,1],[0,1,0]])             #类别数是3\n",
    "W = np.array([[0.1,0.2,0.3],[0.4,0.2,0.8]])  #  2X3矩阵\n",
    "\n",
    "reg = 0.2;\n",
    "print(gradient_softmax_onehot(W,X,y,reg))\n",
    "print(loss_softmax_onehot(W,X,y,reg))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6.7 softmax回归的梯度下降法的实现\n",
    "\n",
    "在此基础上的梯度下降算法代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent_softmax(w,X, y, reg=0., alpha=0.01, iterations=100,gamma = 0.8,epsilon=1e-8):\n",
    "    X = np.hstack((np.ones((X.shape[0], 1), dtype=X.dtype),X))  #添加一列特征1  \n",
    "    v= np.zeros_like(w)\n",
    "    #losses = []\n",
    "    w_history=[] \n",
    "    for i in range(0,iterations):\n",
    "        gradient = gradient_softmax(w,X,y,reg)\n",
    "        if np.max(np.abs(gradient))<epsilon:\n",
    "            print(\"gradient is small enough!\")\n",
    "            print(\"iterated num is :\",i)\n",
    "            break            \n",
    "        \n",
    "        w = w - (alpha * gradient)\n",
    "        #v = gamma*v+alpha* gradientz\n",
    "        #w= w-v\n",
    "        #losses.append(loss)\n",
    "        w_history.append(w)\n",
    "    return w_history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对一组样本(X,y)，下面的辅助函数计算历史记录w_history里的每个模型参数对应的模型损失："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_loss_history(w_history,X,y,reg=0.,OneHot=False):\n",
    "    loss_history=[]\n",
    "    X = np.hstack((np.ones((X.shape[0], 1), dtype=X.dtype),X))\n",
    "    if OneHot:\n",
    "        for w in w_history:\n",
    "            loss_history.append(loss_softmax_onthot(w,X,y,reg))\n",
    "    else: \n",
    "        for w in w_history:\n",
    "            loss_history.append(loss_softmax(w,X,y,reg))\n",
    "    return loss_history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.6.8  spiral数据集的softmax回归"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w:  [[-0.05920585  0.00607797  0.05312788]\n",
      " [ 1.1810134   1.02222379 -2.20323719]\n",
      " [-2.39215667  2.77296386 -0.3808072 ]]\n",
      "[1.0681234262911812, 0.8321410771358214, 0.788734093154218, 0.7757732070848885, 0.7709659484297107, 0.7689671616291273, 0.7680780120185511, 0.76766521456332, 0.7674681036477377, 0.7673721797769464]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x233bdfd7b20>]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAaUElEQVR4nO3de5BcdZ338feHmQyQCyQxQ4Ak5MKGS2QNZIfIRRHlARJKyOqKC2uJy+NuihJYWX125dFd1yrcLcVVa10sKdyHEq3l5gUNEg0ugqyikgkmhACBIYIZc8VwCcUlTPJ9/vid2XSGnpmemZ4+nXM+r6qu030u6c+c6Xz6zK9PdysiMDOz4jog7wBmZja6XPRmZgXnojczKzgXvZlZwbnozcwKrjXvANVMmTIlZs2alXcMM7P9xqpVq56NiPZqy5qy6GfNmkVnZ2feMczM9huSnulvmYduzMwKzkVvZlZwLnozs4Jz0ZuZFZyL3sys4Fz0ZmYF56I3Myu44hR9BHz2s7BiRd5JzMyaSnGKXoIvfAGWL887iZlZUylO0QMcdhhs25Z3CjOzpuKiNzMrOBe9mVnBuejNzAqueEX/7LOwe3feSczMmkbxin7PHtixI+8kZmZNo3hFDx6+MTOr4KI3Myu4QYte0o2Stkl6pJ/lkvQVSV2SHpa0oGLZ05LWSlotafS/MspFb2b2BrUc0X8DWDTA8sXA3OyyFPhan+XvjIgTI6JjWAmHwkVvZvYGgxZ9RNwPDPTq5hLgm5H8Cpgo6Yh6BRySyZPhgANc9GZmFeoxRj8N2FhxuzubBxDA3ZJWSVo60D8iaamkTkmd27dvH16SlhaYMsVFb2ZWoR5FryrzIpueHhELSMM7l0s6o79/JCJuiIiOiOhob28ffhq/acrMbB/1KPpuYEbF7enAJoCI6J1uA+4AFtbh/gbmojcz20c9in4ZcEl29s0pwAsRsVnSOEkTACSNA84Bqp65U1cuejOzfbQOtoKkW4AzgSmSuoF/AsYARMT1wHLgPKALeBm4NNt0KnCHpN77uTkiflzn/G/kojcz28egRR8RFw+yPIDLq8zfAMwffrRhOuwwePFFePVVOOight+9mVmzKdY7YwGmTk3TrVvzzWFm1iSKV/RHZKfwb96cbw4zsyZRvKI//PA03bIl3xxmZk2ieEXvI3ozs30Ur+gPOwwkF72ZWaZ4Rd/amsreRW9mBhSx6CGN03uM3swMKGrRH3GEj+jNzDIuejOzgitu0W/dmr4o3Mys5IpZ9IcfDrt3w7PP5p3EzCx3xSx6n0tvZvY/XPRmZgVX7KL3KZZmZgUt+t7Pu/ERvZlZQYt+7Fg45BAXvZkZRS16gGnTYNOmvFOYmeWuuEU/fTp0d+edwswsdy56M7OCK27RT5uWxuh7evJOYmaWq+IW/fTp6SMQ/N2xZlZyxS568PCNmZWei97MrOCKW/TTpqWpi97MSm7Qopd0o6Rtkh7pZ7kkfUVSl6SHJS2oWLZI0vps2dX1DD6oN70JDjwQfv/7ht6tmVmzqeWI/hvAogGWLwbmZpelwNcAJLUAX82WzwMuljRvJGGHRPIplmZm1FD0EXE/sGOAVZYA34zkV8BESUcAC4GuiNgQEbuAW7N1G8dFb2ZWlzH6acDGitvd2bz+5lclaamkTkmd27dvr0MsXPRmZtSn6FVlXgwwv6qIuCEiOiKio729vQ6xSC/I/v73EP3erZlZ4dWj6LuBGRW3pwObBpjfONOnw65dUK+/EMzM9kP1KPplwCXZ2TenAC9ExGZgJTBX0mxJbcBF2bqNMyN7ntm4ceD1zMwKrHWwFSTdApwJTJHUDfwTMAYgIq4HlgPnAV3Ay8Cl2bIeSVcAK4AW4MaIWDcKP0P/Zs5M02eegT/5k4betZlZsxi06CPi4kGWB3B5P8uWk54I8tFb9E8/nVsEM7O8FfedsQCTJsH48emI3syspIpd9FI6qnfRm1mJFbvoAWbNctGbWakVv+h9RG9mJVeOon/uOdi5M+8kZma5KEfRg4/qzay0ylP0PsXSzEqqPEXvI3ozK6niF/3UqdDW5qI3s9IqftEfcEA6qvfQjZmVVPGLHmD2bNiwIe8UZma5KEfRH300PPVU3inMzHJRnqJ//vl0Pr2ZWcmUp+jBR/VmVkrlKPo5c9LURW9mJVSuovcLsmZWQuUo+vHj0/n0PqI3sxIqR9GDz7wxs9IqT9HPmeOiN7NSKk/RH300dHfDa6/lncTMrKHKVfQR/igEMyud8hT93Llp+sQT+eYwM2uw8hT9McekqYvezEqmPEU/eTJMmQLr1+edxMysoWoqekmLJK2X1CXp6irLJ0m6Q9LDkh6UdELFsqclrZW0WlJnPcMP2bHHuujNrHQGLXpJLcBXgcXAPOBiSfP6rPZJYHVEvAW4BPi3PsvfGREnRkRHHTIPn4vezEqoliP6hUBXRGyIiF3ArcCSPuvMA+4BiIjHgVmSptY1aT0ccwxs3QovvJB3EjOzhqml6KcBGytud2fzKq0B3gsgaSEwE5ieLQvgbkmrJC3t704kLZXUKalz+/btteYfmmOPTVO/IGtmJVJL0avKvOhz+3PAJEmrgSuB3wA92bLTI2IBaejncklnVLuTiLghIjoioqO9vb229EPVW/QevjGzEmmtYZ1uYEbF7enApsoVIuJF4FIASQJ+m12IiE3ZdJukO0hDQfePOPlwzJmTvkPWR/RmViK1HNGvBOZKmi2pDbgIWFa5gqSJ2TKAvwLuj4gXJY2TNCFbZxxwDvBI/eIP0YEHpu+P9RG9mZXIoEf0EdEj6QpgBdAC3BgR6yRdli2/Hjge+Kak3cCjwIezzacCd6SDfFqBmyPix/X/MYbg+OPh0UdzjWBm1ki1DN0QEcuB5X3mXV9x/ZfA3CrbbQDmjzBjfb35zbBiBbz+OowZk3caM7NRV553xvZ685tTyXd15Z3EzKwhyln0AOvW5ZvDzKxBylf0xx0HkovezEqjfEU/dmw6zdJFb2YlUb6ihzR846I3s5Iob9E/8QTs2pV3EjOzUVfeou/pgSefzDuJmdmoK2fR//Efp+nDD+ebw8ysAcpZ9McdB21tsHp13knMzEZdOYu+rS0N36xZk3cSM7NRV86iB5g/30f0ZlYK5S36E09M3za1ZUveSczMRlW5ix58VG9mhVfeon/LW9LU4/RmVnDlLfpJk2DmTB/Rm1nhlbfoAU46CVatyjuFmdmoKnfRd3Skd8c+/3zeSczMRk25i/7kk9PUR/VmVmDlLvqOjjRduTLfHGZmo6jcRT95Mhx9tIvezAqt3EUP6ajeRW9mBeaiP/lk2LgxvUvWzKyAXPS9L8g++GC+OczMRomLvqMDWlvhl7/MO4mZ2aioqeglLZK0XlKXpKurLJ8k6Q5JD0t6UNIJtW6bu7Fj0xunHngg7yRmZqNi0KKX1AJ8FVgMzAMuljSvz2qfBFZHxFuAS4B/G8K2+TvttDR08/rreScxM6u7Wo7oFwJdEbEhInYBtwJL+qwzD7gHICIeB2ZJmlrjtvk7/XR45RV/7o2ZFVItRT8N2FhxuzubV2kN8F4ASQuBmcD0Grcl226ppE5Jndu3b68tfb2cemqaevjGzAqolqJXlXnR5/bngEmSVgNXAr8BemrcNs2MuCEiOiKio729vYZYdTR9Ohx1lIvezAqptYZ1uoEZFbenA5sqV4iIF4FLASQJ+G12GTvYtk3jbW+De++FCFC15yczs/1TLUf0K4G5kmZLagMuApZVriBpYrYM4K+A+7PyH3TbpvGOd8DmzenTLM3MCmTQoo+IHuAKYAXwGHB7RKyTdJmky7LVjgfWSXqcdIbNRwfatv4/Rh2ceWaa3ntvrjHMzOpNEVWHzHPV0dERnZ2djb3TiDRWf8YZcMstjb1vM7MRkrQqIjqqLfM7Y3tJ6aj+vvtS6ZuZFYSLvtKZZ8KWLbB+fd5JzMzqxkVf6Z3vTNN77sk3h5lZHbnoKx19NMyaBXffnXcSM7O6cdFXkuDcc+GnP4Vdu/JOY2ZWFy76vs49F156CX71q7yTmJnVhYu+r3e9C1paYMWKvJOYmdWFi76vQw+FU05x0ZtZYbjoq1m8GFatSh+JYGa2n3PRV3PBBWn6wx/mm8PMrA5c9NWccEI6zfLOO/NOYmY2Yi76aiQ4/3z4yU/g5ZfzTmNmNiIu+v5ccAG8+ir813/lncTMbERc9P054wyYOBG++928k5iZjYiLvj9tbbBkCfzgB/Daa3mnMTMbNhf9QC68EF54wcM3ZrZfc9EP5Oyz0xuovv3tvJOYmQ2bi34gvcM33/9+emHWzGw/5KIfzF/8RRq+Wb487yRmZsPioh/MWWfB4YfDt76VdxIzs2Fx0Q+mtTUd1d91F/zhD3mnMTMbMhd9LT74QXj9dbjttryTmJkNmYu+FvPnp8t//EfeSczMhqymope0SNJ6SV2Srq6y/FBJd0paI2mdpEsrlj0taa2k1ZI66xm+YST467+G3/wmfXyxmdl+ZNCil9QCfBVYDMwDLpY0r89qlwOPRsR84Ezgi5LaKpa/MyJOjIiO+sTOwQc+AAcfDF//et5JzMyGpJYj+oVAV0RsiIhdwK3Akj7rBDBBkoDxwA6gp65J8zZxYnqn7M03w86deacxM6tZLUU/DdhYcbs7m1fpOuB4YBOwFvhoROzJlgVwt6RVkpaOMG++PvKRVPI33ZR3EjOzmtVS9KoyL/rcPhdYDRwJnAhcJ+mQbNnpEbGANPRzuaQzqt6JtFRSp6TO7du315a+0d761nT593+HPXsGX9/MrAnUUvTdwIyK29NJR+6VLgW+F0kX8FvgOICI2JRNtwF3kIaC3iAiboiIjojoaG9vH9pP0Uh/8zfwxBP+8nAz22/UUvQrgbmSZmcvsF4ELOuzzu+AswAkTQWOBTZIGidpQjZ/HHAO8Ei9wufife+DI4+Ea6/NO4mZWU0GLfqI6AGuAFYAjwG3R8Q6SZdJuixb7RrgNElrgXuAT0TEs8BU4OeS1gAPAndFxI9H4wdpmLY2+Lu/g/vug5//PO80ZmaDUkTf4fb8dXR0RGdnE59y//LL6cvDFyyAH+/fz1tmVgySVvV3CrvfGTscY8fCxz+exukffDDvNGZmA3LRD9dHPgKTJ8M//3PeSczMBuSiH64JE+Cqq2DZMlizJu80Zmb9ctGPxJVXwiGHwD/+Y95JzMz65aIfiYkT4ZOfhDvv9BeIm1nTctGP1FVXwZw58Ld/Cz3F+ngfMysGF/1IHXggfOEL8Mgj/rx6M2tKLvp6eM974B3vSGP1zz+fdxozs3246OtBgi9/OX2n7Gc+k3caM7N9uOjr5aST4LLL4CtfgV/8Iu80Zmb/w0VfT5//PBx1FFx6KbzySt5pzMwAF319TZgAN94ITz4J//APeacxMwNc9PX3rnelIZwvf9lDOGbWFFz0o+Haa2HmTPjgB+G55/JOY2Yl56IfDRMmwC23QHc3XHKJv3bQzHLloh8tp5wCX/oS/PCH8LnP5Z3GzErMRT+aLr8cLr44vZHqnnvyTmNmJeWiH00S3HADHHccXHghPPZY3onMrIRc9KNt/Pg0fNPWBosWwaZNeScys5Jx0TfC7Nnwox/Bjh2weDG88ELeicysRFz0jXLSSfC978Gjj8IFF8BLL+WdyMxKwkXfSGefDd/6Fvz85+nIfufOvBOZWQm46BvtoovSOfa//CWcc44/1tjMRp2LPg/vfz98+9uwahWcdZZfoDWzUVVT0UtaJGm9pC5JV1dZfqikOyWtkbRO0qW1blta73kPfP/7sH49vPWtsGZN3onMrKAGLXpJLcBXgcXAPOBiSfP6rHY58GhEzAfOBL4oqa3GbcvrvPPgv/8bIuBtb4O77so7kZkVUC1H9AuBrojYEBG7gFuBJX3WCWCCJAHjgR1AT43blttJJ8Gvfw1z58L558OnPw27d+edyswKpJainwZsrLjdnc2rdB1wPLAJWAt8NCL21LgtAJKWSuqU1Ll9+/Ya4xfEtGnpyP4v/xKuucbj9mZWV7UUvarMiz63zwVWA0cCJwLXSTqkxm3TzIgbIqIjIjra29triFUw48alLy256SZYuRLmz4fbb0/DOmZmI1BL0XcDMypuTycduVe6FPheJF3Ab4HjatzWKl1yCXR2pnfT/vmfw/veB1u35p3KzPZjtRT9SmCupNmS2oCLgGV91vkdcBaApKnAscCGGre1vo4/Hh54IH0H7V13pdvXXQc9PXknM7P90KBFHxE9wBXACuAx4PaIWCfpMkmXZatdA5wmaS1wD/CJiHi2v21H4wcpnNZW+Pu/h9WrYcECuPLK9MLtT3+adzIz288omnAMuKOjIzo7O/OO0Twi0jn3H/sYPP10ekftNdfAwoV5JzOzJiFpVUR0VFvmd8buD6T0BqvHHoN//Vd46KH0JqslS9IQj5nZAFz0+5ODDoKPfxw2bIDPfjadknn66elrC2+/3WP4ZlaVi35/NGECfOpTsHFjepH22WfTGTp/9Edw7bWweXPeCc2sibjo92fjxqXvpV2/Po3hz5wJn/gEzJgB7343fOc78Npreac0s5y56IugpSWN1//sZ/D443vP1rnwQjjySPjwh9Npmi59s1Jy0RfNscfCv/wLPPMMrFiRvuDkO99JR/jt7enz8G+9NQ33mFkptOYdwEZJS0s6DfOcc9KR/L33pq8y/MEP4Lbb0pk8Cxakb706+2w49VQ4+OC8U5vZKPB59GWze3f6iIWf/ATuvjt901VPD4wZk4r/1FPhtNPSZVrVz58zsyY00Hn0Lvqy27kT7rsPfvGLdE7+ypXw6qtp2VFHwcknpw9Ymz8fTjwxvdCrap9VZ2Z5ctFb7XbtSi/kPvBAujz0EDz11N7lEyem0j/hhPR6wDHHpMtRR6XhIjPLhYveRmbnTli7Nn3d4Zo16Yng0UfT/F5tbek8/t7inzUrlX/v5dBDc4tvVgYDFb1fjLXBTZiwd9y+VwRs2wZPPPHGy/Ll6S+DSocckgp/xox0OfxwmDp177T3+vjxjf3ZzErARW/DI+0t6Le/fd9lu3enz9D/3e/2XjZu3Hu9szOd3lntr8mxY/f+u296E0yalC6TJ7/xeu904kQ48EC/dmDWDxe91V9LS3qj1pFHps/hqaanJ5X9li3pSaH3Unl7y5Y0RPTcc/D88wPfZ2tr+stj/Pg0Hez6wQfvvRx00MC3Dz44nZXkJxLbT7noLR+trWmo5vDDa1t/92544QXYsSMV/44de68//3x6veCll9K08vrWrfvO7zukVKsDDtj7BNDWli5jxox8OmZM2hctLdUvAy2rdZ3e5QcckJ6spL3Xa50OZ5veqeXORW/7h5aWNFQzefLI/p1du1Lxv/oqvPLK3stQbu/alS6vv54uvdd7pzt3Vp9fbdqEJ0OMilqfDHqv9zcd7rJGrjOS7adMgfvvp95c9FYubW0jf7Kop9270zDW7t3VLwMtG+o6EbBnT5pWXq91OpxtBtt2z569+6L3Sa+/6XCXNXKdkW4/SmenuejN8tQ7vGI2ivyhZmZmBeeiNzMrOBe9mVnBuejNzArORW9mVnAuejOzgnPRm5kVnIvezKzgmvLz6CVtB54Z5uZTgGb85mvnGrpmzeZcQ+NcQzecbDMjor3agqYs+pGQ1Nnfh+/nybmGrlmzOdfQONfQ1Tubh27MzArORW9mVnBFLPob8g7QD+caumbN5lxD41xDV9dshRujNzOzfRXxiN7MzCq46M3MCq4wRS9pkaT1krokXZ1jjhmS7pX0mKR1kj6azf+MpN9LWp1dzssp39OS1mYZOrN5kyX9RNKT2XRSgzMdW7FfVkt6UdJVeewzSTdK2ibpkYp5/e4fSf83e8ytl3RuDtm+IOlxSQ9LukPSxGz+LEmvVOy76xucq9/fXaP2WT+5bqvI9LSk1dn8Ru6v/jpi9B5nEbHfX4AW4ClgDtAGrAHm5ZTlCGBBdn0C8AQwD/gM8H+aYF89DUzpM+9a4Ors+tXA53P+XW4BZuaxz4AzgAXAI4Ptn+z3ugY4EJidPQZbGpztHKA1u/75imyzKtfLYZ9V/d01cp9Vy9Vn+ReBT+ewv/rriFF7nBXliH4h0BURGyJiF3ArsCSPIBGxOSIeyq7vBB4DpuWRZQiWADdl128C/jTHLGcBT0XEcN8ZPSIRcT+wo8/s/vbPEuDWiHgtIn4LdJEeiw3LFhF3R0RPdvNXwPTRuv+h5BpAw/bZQLkkCXg/cMto3PdABuiIUXucFaXopwEbK2530wTlKmkWcBLw62zWFdmf2Dc2enikQgB3S1olaWk2b2pEbIb0IAQOyykbwEXs+5+vGfZZf/un2R53/xv4UcXt2ZJ+I+lnkt6eQ55qv7tm2WdvB7ZGxJMV8xq+v/p0xKg9zopS9KoyL9fzRiWNB74LXBURLwJfA44GTgQ2k/5szMPpEbEAWAxcLumMnHK8gaQ24ALg29msZtln/Wmax52kTwE9wH9mszYDR0XEScDHgJslHdLASP397ppln13MvgcUDd9fVTqi31WrzBvSPitK0XcDMypuTwc25ZQFSWNIv8D/jIjvAUTE1ojYHRF7gK8zin/iDyQiNmXTbcAdWY6tko7Ish8BbMsjG+nJ56GI2JplbIp9Rv/7pyked5I+BLwb+EBkg7rZn/l/yK6vIo3rHtOoTAP87nLfZ5JagfcCt/XOa/T+qtYRjOLjrChFvxKYK2l2dlR4EbAsjyDZ2N//Ax6LiC9VzD+iYrX3AI/03bYB2cZJmtB7nfRC3iOkffWhbLUPAT9odLbMPkdZzbDPMv3tn2XARZIOlDQbmAs82MhgkhYBnwAuiIiXK+a3S2rJrs/Jsm1oYK7+fne57zPgfwGPR0R374xG7q/+OoLRfJw14lXmBr2SfR7p1eungE/lmONtpD+rHgZWZ5fzgG8Ba7P5y4Ajcsg2h/Tq/RpgXe9+At4E3AM8mU0n55BtLPAH4NCKeQ3fZ6Qnms3A66QjqQ8PtH+AT2WPufXA4hyydZHGb3sfa9dn6/5Z9jteAzwEnN/gXP3+7hq1z6rlyuZ/A7isz7qN3F/9dcSoPc78EQhmZgVXlKEbMzPrh4vezKzgXPRmZgXnojczKzgXvZlZwbnozcwKzkVvZlZw/x8K+gxf4HZrUwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_spiral,y_spiral = gen_spiral_dataset()\n",
    "X = X_spiral\n",
    "y = y_spiral\n",
    "\n",
    "alpha = 1e-0\n",
    "iterations  =200\n",
    "reg = 1e-3\n",
    "\n",
    "w = np.zeros([X.shape[1]+1,len(np.unique(y))])\n",
    "w_history = gradient_descent_softmax(w,X,y,reg,alpha,iterations)\n",
    "w = w_history[-1]\n",
    "print(\"w: \",w)\n",
    "loss_history = compute_loss_history(w_history,X,y,reg)\n",
    "print(loss_history[:-1:len(loss_history)//10])\n",
    "plt.plot(loss_history, color='r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面函数可以计算训练模型在一批数据(X,y)上的预测准确性："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getAccuracy(w,X,y):\n",
    "    X = np.hstack((np.ones((X.shape[0], 1), dtype=X.dtype),X))  #添加一列特征1 \n",
    "    probs = softmax(np.dot(X,w))\n",
    "    predicts = np.argmax(probs,axis=1) \n",
    "    accuracy = sum(predicts == y)/(float(len(y)))\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对刚才的数据集用getAccuracy()计算训练得到的softmax模型的预测准确度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5133333333333333"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "getAccuracy(w,X_spiral,y_spiral)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面代码绘制出对于这个问题的softmax模型的分类边界："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-1.9181692991988153, 1.901830700801188)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXxcZ33o/8/3nJnRvi+2LNuy7DjxEi9JHMdxQnBCCNkgQKEQWgJcaKCUll8v93cvLW1v29ty4d5y76+5FwIpYWvLWhJImkBWQkhCFu9rvMmbbFmSrXW0zMw55/n9cUayZM2MRtJoJHm+79dLL0szR3MeHUvfec7zfJ/vI8YYlFJKXfqsmW6AUkqp7NCAr5RSOUIDvlJK5QgN+EoplSM04CulVI4IzHQDUikuKzcLGmopCQaQ2d1UpZSaFbZt33POGFOT6LlZHUUXLWzgTx74/8AM8s5riqjPb5zpJiml1Kxm5S0+kfS5bDZkMtbXrKQ01MDj2/p4tuUAMc7PdJOUUmpOmtU9/CFLyyuBSnYfO86RphbeeU2P9vaVUmqCZn0Pf6S1NUv83v52l6+/vFd7+0opNQFzKuCD39tfX7MSSyp4+OUWnm05MNNNUkqpOWHOBfwhQ739I8eCfP3lvZwePDbTTVJKqVltzgZ8GN3bH5rUVUopldicDvhDtLevlFLjuyQCPiTu7eukrlJKXXDJBPwhI3v7D7/cor19pZSKu+QCPmhvXymlErkkA/6QtTVLWF97lfb2lVKKSzzgD9HyDEopNUdKK2SClmdQSuW6nOjhjzRcnkF7+0qpHJMzPfyRtLevlMpFOdfDH0mLsSmlcklOB3zQYmxKqdyR8wF/iJZnUEpd6jIS8EXkWyLSJiJ7kzy/RUS6RWRn/OOvMnHeTNNibEqpS1mmevjfAW4f55jfGGPWxz/+NkPnnRba21dKXYoyEvCNMS8CHZl4rdlCyzMopS412RzDv15EdonIL0RkdbKDROR+EdkqIls7O2c+wGoxNqXUpSJbAX870GCMWQf8H+BnyQ40xjxkjNlgjNlQUVGVpealpr19pdSlICsB3xjTY4wJxz9/EgiKSHU2zp1JWoxNKTWXZSXgi8h8EZH45xvj552zXeSRxdg06Cul5oqMlFYQkR8AW4BqEWkG/isQBDDGfB14H/CHIuIAA8AHjTEmE+eeKUvLK9nd3sOBzh5q684TZHYMPymlVDIZCfjGmHvHef7/Av83E+eaTYqDpRw5NqD1eJRSc4KutJ0CncxVSs0lGvAzQCdzlVJzgQb8DNLevlJqNtOAn2Ha21dKzVYa8KeJ7qOrlJptcnLHq2zRnbWUUrOJ9vCzQHfWUkrNBhrws0R31lJKzTQN+Fl2ca197e0rpbJFA/4M0N6+UmomaMCfQbqzllIqmzTgzzAtz6CUyhYN+LOE7qyllJpuGvBnEe3tK6Wmkwb8WUjLMyilpoMG/FlMe/tKqUzSgD/LaW9fKZUpGvDnCC3GppSaKi2eNodoMTal1FRoD38O0mJsSqnJyEjAF5FviUibiOxN8ryIyAMickREdovI1Zk4by7T8gxKqYnKVA//O8DtKZ6/A1ge/7gfeDBD5815Wp5BKZWujAR8Y8yLQEeKQ+4Bvmd8rwLlIlKXiXOrxAu2lFLqYtkaw68HTo34ujn+2Bgicr+IbBWRrZ2dOjY9EZdKb/90cx/793bS2xOd6aYodUnJVpaOJHjMJDrQGPMQ8BDAytXrEh6jkhvO5Gk/zuPbOrls6QHeWldLkKqZbtq4Ojsi/O8v7eHsmX7sgBCLedz89gXce99lWFaiXyGl1ERkq4ffDCwa8fVC4EyWzp2T5loxNmMM//O/7aL5ZJho1GOg38WJGX79XAvP/KJ5ppun1CUhWwH/MeC+eLbOJqDbGNOSpXPnrNlejM3zDLGYB8Cxo72cPzeI540+JhrxePKxUwm+Wyk1URkZ0hGRHwBbgGoRaQb+KxAEMMZ8HXgSuBM4AvQDH8vEeVV61tYsAZaw89iBWbFgKxyO8b1vHmLba+fwPMOihmJWrC4jEvESHt/bHctyC5W6NGUk4Btj7h3neQP8USbOpSZvfc1Kmro6eHzbiRkb2/c8wxf/cgetZwdwXX+K5uTxMCePh5N+z/wFBdlqnlKXNF1pm2OWllfOaDG23Ts76DgfGQ726XjXexumsUVK5Q4N+DlqpoqxnTzWSyTipn28ZcEVq8qnsUVK5Q4N+DlsqLffdKw0a739yqp8Qnl22seXV+ZRXhGaxhYplTs04KspFWMbGHDo7IjgeekN0Vy7qQbbTi+nPhiy+OgfXI6I5uArlQlaHlkBoxdsPfxyC5ctbePWupVJjw/3xnj4wTfZs7MDEaGgMMDvfewyrttcm/I8efk2/+W/ruMfv7yXcDhGNElmjgj8579cx/IryqbyYymlRtAevholnfIMxhi+/Dc72b2jA8fxc+l7uqM8/LU32bs7VUklX8OSEr7ytU382V+vZ978fC7uwAcCwqYbajXYK5VhGvDVGOMVYzv0ZjdtbQNjMm2iUY9HfpjePICI0LislP/8V+uprsknP98mlGeRl2exuLGY+z5xecZ+HqWUT4d0VFJra5bQ1FXKkWOtHGnayzuvKaI+v5HTp/owiUdiOHtmYELnqKrO58sPXMeb+7pobxtgUUMxjctKcnvcXiJgdYFEweSDW058HaNSU6IBX6WUqBhbzfzCpMXMqmvzJ3wOyxJWrakAKqbW2NlAImB1xoN1Hnjl/r9pf38vBNoAgwgYEwGrB5yFE3sdpRLQgK/SMtTbP3iwhad+fZrBwbG59KE8i3vel8OLpKQPAme5EKyjYIXBqQNTmMYLGAi0I3JhqMx/HQN2GziLRh9rhUEGwQTAKwXST3dVuUnH8FXaGssqOPl4H+feHFvbJpRn8YHfX8o1G2tmoGWzgYFAGyJmeBJaBD94x3vsF0T9uwCrExhxLWWQRFXDRfDvHBgaR3MgeALsNsTuBrsDgsdBJjacpnKP9vBV2lpPdnP+bBiToCzC8stLeds7Fs5Aq2aLGBcC8sUcCJwCtxKsXrD6Ljxln/cf9ypJvG1EAnYr4Ix4Y/H/P0zgLMSWpP86KudowFdpO3X4HK6bOKgdOdZNjPNzYqOV1Fx/qATXnzA1BaQMoDLgB3EckuzpE++hRzFy9sLXIxi7wx/yMXn4N92jh8uMId4OC6xzYA2MeQ2f598JmInPo6jcoAFfDYtFXQ7tPMPJg+coKAqxauNCaheW0dnex9Pf30VnW1/S77Xz/WJsM116eUpkAAJD+/IYQMCEwKkn4einfQ6sbi6M2fsfyRKMUiYeWd3gzgNnPibeBhGDMeKf26n15wjsrtSvM+YuwwG7y//ZTCA+iazVR3OVBnwFQGQgxk+/9hr9vRGcmIcIHNl9lmtuWcrO35wg0p+8Jn0gaLH5bVdghUI8vu0EsJeP31A3x3r7BgItoyZMwWCI+mPkbvWIx934GPzo4JtO0E9EBIzEr68pgFgDWD3+YyYfvBLA8oeFxv0xRvbuYxA8BXgjMn76wa3yA7/KOTppqwDY+nwT4e5BnPgOVMaAE/N445mjOLHU1S0XNFawbM284WJsllTMiW0VfQakH6wOEk+YmngvfuiBwfiEaeKe9mSWDxgDeCOzeAJ+kPfKLgT7+BxBstc3BnBrGPUnbbeP+p7hSWT7PBcPG6ncoD18BcDRPWfxktSod2PJJiNBLKhfVjVqodRQCufj21vBzOLevgxCoAV/GCRV8TcTHxsPQeAsIsmvB6QO+hf3/s3QaYd73I7fJoleOMitSJnW6b9hhMArGv2E1Z+8LdIHphjt8+UW/d9WqQkpq1talsXiK6rHPD6yPMPDL7eMKc+QHYaEgVz6/OGRQDMi7nAqZeox9qEUyqn1jEcO+/jBXiC2CP9P0UDwNEgk3iYT75F3gH025WtiRf07D6L4b2DRpMcPpZASbAL7zJR/JjV3aMBXACxbMx8rQWAXEYJ5gYTB0A5YrLhmARU1RWOfjEunGFvmRf3J1+BR/8Nuwc+iwR+eCZxFrEjawy9+lk0MxunZp0/8cglOPcSWAvEVtDLAyHTL0ecf+/iYY/D8N4zgMQieTHns8Buc1Q+BZlLf4ahLRUYCvojcLiIHReSIiHw+wfNbRKRbRHbGP/4qE+dVmbPhlqUUl+cTCPq/EiL+ZOx1t13Gez99HbWLyrBsQSzBsoWqumJu/d013PjOFeO+9sXF2KY36DsQbAbpHxHU+vzHcMA+d9HE7Pj8tMi8eNpk4qh7ocee7osWj035FCfp4em8OfnHpHnHMvJ7xNFFWzliymP4ImIDXwXeDjQDb4jIY8aY/Rcd+htjzN1TPZ+aHnkFQd7/mes5vKvFT8ssDrHq2oXU1JcC8J5PbmSgL4rreBSV5k2quNnamiXsbofUww1TZHdx8eSmP4wSX506qZ5svEeOgFODGVXrJv68Mw8CbRhz4dxDbwBjxuxNIHFdHDP1nb0mV3POxGv/pFP+wcF/k9IyDnNRJiZtNwJHjDFNACLyQ+Ae4OKAr2a5YMhm1bULWXVt4hWzBUVTD0jFwWmezJXBFNkzk+jZI+AsAOI/uykBJwhWF2ZUNcsQxPLAPoex+v1jvSLAXPga/GDvLCDhnYLJAxPCEBnzJpFOIJ9oOugF4rfLfxU/a0mceHviaZ4y6K/wHU4fDfnrBrSg25ySiYBfD4xMEG4Grktw3PUisgs4A/wnY8y+RC8mIvcD9wPMq6vPQPPUbHLxzloZX6hlAlMIfKMNFy67eOWqyQd3foLvCIJbN3YO1HXitXAC8V580tQZf1zfbsNY4fhjNnh5GGtg1FDUyOGjC3caE+d/nwWmCIj6cwAjF2+ZPH/RV+D06PMThcBpf82A9vbnjEwE/ES/vRf/+m0HGowxYRG5E/gZsDzRixljHgIeAli5ep3OJGVYX0+Eo3vOMtgfo35ZJQsaK2ak9vy0pW56pfHSCONLNOQyPQIjetDjsfw3E3coVTQ+zWa6MHYnfjC24/V3CsHuxhBfoCVRjBUed45i+M4FwAT9ap4AQT9jZ9TdBZHhCqAjDb8ZWj3gXQJlrXNEJgJ+MzCybutC/F78MGNMz4jPnxSRr4lItTHmXAbOr9J0dO9Znv+3fWAMrmPY/cpJautLueujV2MHLFqOd/LKk4c4d6aHQNBm1caFbHz7ZdiB6Unmmug+uuOyeuKLjcb2eJMGdS8Pkypjx+QzM8XILrrmXkU8sJrR7Rm1Ath/zlg947+JOQvA2EAwXg/oPIkzhPzVxomHyQxGImn+PGo2yMRf8hvAchFpFJEQ8EHgsZEHiMh8iXcjRWRj/LznM3BulabIQIzn/20fbszDdfxo6ERdWk91s+ulE5w92cW/f3s77ad7MMavq7PrpRP89GuvTnvbMpO6GQO7fVSGSsIFTiO/9orBXQTOAowXHJVp438u8dWrs0mqSC7g1oJXnHCIZ/hncurjGUJBf5FXoGXcu6LErycZmWhW2TPlHr4xxhGRzwBP4Q/mfcsYs09EPhV//uvA+4A/FBEHGAA+aMxkRx3VZBzb35awl+Y6Hge2NnPiYDuuMzbPvKO1j9/+8iCbbrscSbLLVSYk2lnrrXW1KYZ5jD+ROByoUq+WHRv8LT84gp+d4jTESyx0Xahh41YwPFk7l7g1IBEMTrwA29DjQ3cJ8X6e1Q0yMO4QUEpeyVRbq7JIZnPcXbl6nfnWj5+a6WZcEva+eorf/uJQwqCeXxgkGnGSllYAKKnI553/4RpKK9NJ3Zuapq4OemKtYAaH99EdzfgZI1YfI4P8RMbijSG+6OlSXXto/BXF1tCOWCWMmVwNnESs1CmyyeY5LoQNG2ILSb3n7tCktZ1yLYPKDCtv8TZjzIaEz2W7MWpmLLysMulzkcFYymAPEO4a5Inv7CAbHYSLF2rt7X4TEzjmryC1W8Bqh/jk5Mjhm2TDGMnN3s7O1Im/uMutjtfpSZRJk/jnHxr6MZ4/f5EszXVooReB1iRtMP6cSvCEP/EbOB1fAZy88qqaXhrwc0R5dRErrlkwvJJ2JJNGxQBjoL83QvvpnvEPzpC1NUu4Z80CVlQGsCwXEdfv1dvJJyVHBvjUwT5Azv/6m6Ik1yg+zu+Mv4OZv1J3kOHSFSNZ3WD1jK4LRCye+nkpv9nOXjn+G59bbnznCq65eemkx+I9z9DXk72sjGAgxvzKToIjOqepSgaMXOF6YZy+JL6JyMjnxM8tz/WhBbcCCFx0fcSfzB5ae+Ale1MYSRgdwA1IOGEZi+G7Ai3lMCM04OeQrnP9bPtVE8abXO/KdTy2Pn804TzAdCgt7GeiTR1d+90DOwxeCcYrwJiAn7vu1KdZRuBSZ/uVOt0KjBfyh3Dc2guT2RBP+7THHxqzOxkO+vZ5CLSmnlMRrdA5EzTg55Btv2rCmWKw7j7Xz5E9yUv1ZpLr2UymFz66jo7x95x16vwNvt0FY1fO5jTb30DdWewP4XgljL7mgXj55uT8AnW9w9U+sbrHz/zxtCTDTNCAn0NaT3ZPeejUiXkc3pWdgN/dV5S5oV5dIDQF6YSJ+KpbK/VQjT9kVMCcTHe9BGjAzyHFZeP3quzA0JhIqmOy82tjjHDg5CIc18J1ZeIliEfRX/XJs/wSDCkMF6czyX9x/MVuZX69ITUj9K8gh6x/y5KEWTojiWVx8++s4l2f2JDw2EDIZuU1C6ariWOEBwrZdmg5fYP+MMxE695cKEesPcopcWv8VM0kb7j+qtvipHMjfrAvj88J5Phk+QzSgJ9DGlbUsOGWZdgBK2ngdKIuzUc7KCwOkVcwulcXCFo0rqyhYUV2yw14nkVRQeKyx6kMZ+o4dWQjyPQcbePUEzvo3H96RBsMXW+eoXPvKTw3O5Pd08LEJ7u9gjF3WsMlFrz4HrlOXfzNQS5kRZl8v+CbmlG6iXmOWX/TElZurOeX/7yLluOdY563LCGUH+TRr79OZHB0brVlW9zwzivwXINlMa2lFi42ueX/Jr4KdHp7985AlBc+9FVafrUfKxTAi7lUrlvMVX/5bl7+9HeItPeCBYHCPN7y8B9Q//Y1Y16j+1ALBx58jp4jZ6ndfDkrPrGF/JrSaW33hJl8cOvBNWB1Y6xe/3GvxB+qGa7AWehPkFthwPW/7+LdvdSM0NIKOersyS7+/VvbcGKje5120GLFNfW8ubV5uMja8HO2YIdsYoMOdsBi5bUL2fSO5VkZ07+ysYnigrFVLVOVODYG9jQ7VJblZbbm/kVe/uTDNP3oVdzBCytIJWiBB+aiXr2VF+Bdr/0t5SsuDIsdePA5Xv9P/zp8rNgWweJ8bvrep2j77WGcgSiL7lzP/JtWzEgpazW3pCqtoAE/h+1++QSvPX0EyxJ/Jb5neOu7V/Hm9jOcPtox7vfbAYtFy6u4/ffXT3tbC/IirFvWNOFhHc+D/S0OLY4ZpxjbxAy293DqyV3Eegd44/M/xEwg3bVi7SLueuEvOPD1Z9nxdz/D609SasDy9xA2rkegMI+6m1dx84/+GMuezSOxbvxDVzLPlFQBX4d0ctiV1y+mtLKQM8c6KK8pYsmKGsLdg+QXBtPaRcl1PE4dPk/3+X7KqqZvIVMw4LB4XpufoTnBTUssC66YH+DsITK2w9bBh1/g9c/9K8YYvEjyjceT6dx9ih81fJZY72DqAz0zvEjO6YvQ8qv9NH3/FS778I2TafY088Buixe0i3PL/Rx/HcqZNTTg56hzLb08+d0dxKJ+wHJiHr957AB2wMJ46ac/WrZwrqV32gK+YLiy8TihYIzJThkEbdi0JJ+jrQVpll4eLdLZx2B7D8YYtv3FTzj1+I7JNWSEcYN9Ak5fhIMPvzA7A77dClb/6G0Q7S5A4kFfzQYa8HOQ63g8/q1tRBIMJTjRCWaSGCgum76VqxUlvQRsJ+1gn2w/22DAYW3NcmAJO48d4EjT+L39aM8AL/3BN2n+5S7EEtyBma/y6A6kLmc8M5wxwR7iO2LZXfEa/NrLnw10kC0HnTjYjpeBejhiCUVl+dQunL5sksL8QQJ2+vNMySZvA7ZD0PbvZtbXrKQ01MDj2/p4tuUAsSSbrz3/uw9w6okdeBFnVgR7Oz9I4wc2zXQzxpIYyQO6h5ZDnj004Oeg/t4o3iQLqI1UWVvE3R+7elozRyKxEK479vUnkmswVGGzYX4LtuUX7VpaXsn62qs4cizIwy+3jNlWsftQC60vH5rQZGymiX3h57YLghQtqmLFH9w8Y+1JygRJWQMj2JL6eZU1OqSTg2rqSyec7TKSZQtXblrE5juvyFyjkjjfXcqSea0YY6bWZoHqsjBVpYdo7azg+Nl5gLC+ZiVNXR08vu3EqLH9zgOnMbGZreho4pvS2AVB6t+xlhse/BjBkoIZbVNifhVSY/Ul2AQdjIlB8Dh+Tr4dX/VsgymKL9bS4Z5s0R5+DqpdWErtwjI/HXOCxBIqa4u59tbLpqFlY3nGYt/xBqJOYFSvfjLBX8TP2plf2cnqJcepLe/Estzh3n7TsdLh3n6wKF53yLIgL3UdmYxJ8v/hDsQ48/QeXvjQVzHe2DuOSEeY3f/wBM+86yv89k++R+e+5uluaYJG+m+gyYi4/p2W5YI1gNhhP6sn0Iz2/rMnI3n4InI78I/4+6h90xjzpYuel/jzdwL9wEeNMdvHe13Nw58+0YjDd/7+hZRbG0q8O2A8P9Dn5QfYdPtylq+ry1oBtSHLFjRTU96bcOHVZHv+rgsGi33HGuiP+BPPQ/vpVv3qDc5Qz45d8cVQAkEnggxG8Wybwr4eFh/aQ1X76VSnAGDh3etpeXbfqIVZwwSsUJBgSR6Rc+Gxz48QKMpjy/f/iIXvWDv8WPjEOR6/4W9w+gZxB2KIbWGFAtz40MdpfP91E7gaGRA4jYxTLfNixgi4VfFtGBVdx+jfs3tKL1F86x9PXx6+iNjAV4G3A83AGyLymDFm/4jD7gCWxz+uAx6M/6tmSGdrOD72niTgC7z/M5s4vOss0UGHJStrWbisMqvlFEayrMSBfeR6gYkGftsGYzxWN56gtz+fYMBhve0RtAx/+ZOFnDl9YUjHGIjaeVAYAhG68wvZe20V888eZ96JI5R1tGEPbc1lDHZRHst+7wY2/N37CRSEOPqDV3jlD7+NBGzA4MVcGt6zgar1DdRedxkn/30H+x94Ci/FMJLTF+Hk49tHBfzX/uO/EO0ID+frG9fDHYjy8qe+xaK71hMozGLdebcCI4MTKoMhYjBWT1oB3z34m6m0bk6ItHUQaSnBLFgxLa+fiTH8jcARY0wTgIj8ELgHGBnw7wG+Z/zbiVdFpFxE6owxLRk4v5oEzzNYtuAmWTe0+LIqKueVcN1tJdltWBLne0qpKA5jX5Sx47rCybYaFs9rw57kMI9teZQX9w+/YezeHqGtNUngHfGuYgJBWhYup2PZFeQH4f7fnceq25Yj1ti7n2X3bmbRnes5/fQePMel/tYrR9XKKZhfzoEHn4VU8wa2ECy+kAJrjKH5qT0JdzAT8Tjzw0eov2H8fWkzSSoKsOrLJjaRH+nDPZI6mEfaOnDPxXBil3ZOv1lwLUxjMdpMBPx64NSIr5sZ23tPdEw9oAF/htQuLEv6nFjCjfeszGJrxtfRU0K4soDi/IHhoO+6Qt9gPq0dlSye185kx4Ivjk0nmqI4E1hAG4l4RCLwzUfb+MrGYNK7oBDQ+Pah7QPPQ9eFdNCSCrj5offx4qcfwY06uINjGyBAtL2Vs0/+innXN8TbbhL+1MaFgWMuPcXZTomMUXpnIYHK9ArWGcejf1+YyJvjtXP6er25JBMBP9Fv98W/g+kc4x8ocj9wP8C8uvqptUwlZQcstrxnFb/66T5cxxseFrFti7v/w9WUVlzIBjm6p5XXnz1Cb+cAxWX5XHPLUq64Kns18X3CgeOLqSnvpqa8C4D27jLau8oxCN3hIipKwlPK5BlSWW2nVVriYt2dMXb8eA+Xz5tckC2z4a6v3sK5gx0cevwo5w6cxx0qbmf8WkeH/3UHh3+wg0DIxnjxzKWEI3NC5c1vwYSyNOE8wuBZh6KKaNI1EeC/yXoOeP02g+EFsEAzdbIhEwG/GRi56eVC4MwkjgHAGPMQ8BD4k7YZaJ9KYtma+ZRVF7HnlZN0n+9n3uIy1ly/eNTK2f2vN/PKkweHq2r2dAzwm58fYDAcZd1blmS1vQahrauctq6x470nWmspLerHtrwpB/0N1xfwnQc7iU5wV0QPi+9vreKdb6lhzdLiSa9PqKiD67ZA1+HTND+7k6ZHX/Ej5dBfgwfO4OihH7EsjOf5k7YBm/X/+X3YMxDsASJnbULzLYIVif8v3H7wBiwibQGirXbKXbJUZmUi4L8BLBeRRuA08EHgQxcd8xjwmfj4/nVAt47fzw7VdSXc/DurEz7nuR6vPX14TAllJ+ax9fkmVm9aRGBoonKGDUbz2H20kSuXHiMUmNpiqbw8iz//Yg1/+/+2kyALMqXzPQ7/+ssWGuoKCA+4eJ7hqstLeev6cvLzJnatypfX07H3RFrHGs/DLgiBZyi7vJ6CmpnMehH63gxRvmlwzL29CNj50L0tBM7Q7YnKlikHfGOMIyKfAZ7CT8v8ljFmn4h8Kv7814En8VMyj+CnZX5squdV08N1PY7ta6NpXxuCIZqiGmRPxwCV84qz2LrUIrEQh08tZGXDKSxrajeHjcvy+OyfV/KPX+yYcNCPuXCk+UJ64nNbO9h2sIfP3dtA3jhbTAK40Rgnf7mN08/vYvB8L6S5Knqozk7HnuO88rl/YuPf3UfNVcsm1vgMsfIMxr2Q2jv6Sai8cRAvBv1HgkRbZ+ZOJBdlZKWtMeZJ/KA+8rGvj/jcAH+UiXOp6ePEXB775lY62vpwoqlXmXqeIb9o9B+q63pgsrfJeSI9/UXsO76YhnmtFMe3RRzK1fc8wbLMmNz9ZCMvq9bmEwhBdOKFLUdxXEN32OG1fd3ctL4i5bFuNMZLn/0G4ZPtuJHJT7i6kRh7HniMW779p5N+jalw+6zEwR6G5x3sPCheESMMGvSzREsrqGH7X2/mfGv4wkRhCnVLyiks9nO8w12DvPjzA75LupMAACAASURBVJw6ch6MYd6iMm5696oZ6/2HBwrZd9yvgikYKkp6KciLMhAN0dlbTCjgEAw4VJb0UFI4iAGKCwb8lbgCnjG4rsE5cZpP39PA//23kzhTrLIQcwy7j4bHDfinntkx5WA/pO/MeZyBCIGCLObixxlHGDwdIL/eQVKMZIkNhcscDfhZogFfDTu4/UxawR5gy3tXARCLujzy4Gv0hy+U7T17sptHv/E6H/js5mktnZwOg9DRWwq9Fx6LxEJEYiHCAxdq+OeHItRVdYDdRTQSpnJrM5H8a1g8D/7uk5fx3Bsd7DkaxvU82rsmvukJ+D3bn73YRt+gy6olRaxZVkLgosUDzc/tzEiwBz+91grO3J94/5EgnmMobHRTTqRbeQbE6ORtFmjAV8PSHfW2bYvC4jw812PXS8fp7xtbo911PPa8cpLr77g8s42cJoPRPJ47kEd3VxsfXtBFJP+a4edCAYs7rq/mjuurAfjaI6dGjdGnwxI4fmaAptMD/l67R8I8t7WTP37fIvJC/thHLDxAz5HM5DKIbVF342qswExOqgtev41xXSRFpDEOWk4nSzTgq2GXr6/jjWeP4o5TEnjBsgp+/s2ttJ7qTvqH6rmGU4fPs+kdZsbKMUxEU1cH3V17uKOhl2jb2D16Hcdj15EwB0/2UVI48SBqzOi516hjaOuM8qvtHdy+yX8j2fm/Hs1Y776gtow1f3JPRl5rOhkXBk4F0Gyd7NCAr4atvm4RR3adpbO9L2XQP9PUgeuM3yXraA3zT3/9HGs2L2bDLcsIhmZHCmciPbFWls4boGxnEHPRmrKBiMsDPz5FZzhGNGYmtdVioqvluIatB3q4fVM1zkCU1t8ewLhTr78fKMzj5m//KXZg5v+8Yx12ygVYkRabweM6fp8tWh5ZDQuGbN7zqY1cf8fypB0uO2jhpqiweTHPNez6zQl+8sBvicyCXaMu1tTVwc7Tr+J1neEt3eGEy/effv0857v9YA9pZ0mmZei1nMEIU+7lWoJdEGLTlz42K4I9+JO3fQeDGNevuuo/Bl4Mul7Lp+9QHtq7zx4N+GoUO2Bx5abFXHndIgIX5YwHghZ5BcFJjbf2dg2w/dfHxj8wi3a2H6CrfQe3LzjHvWGHQbk24XHbD/biZDLKx9kWrF/uF6fLKy8mVDrFzU08Q6AgRFHd7CowFjkbpHtrPpEzNpE2i74jQbp+W4DXr+En2/SKq4RuuOsKrt6ylLwCv6dYUpHPW9+7ipoFk6ueaQwc2X2WcNcgR3afpfno+YxsszglZpCPlnRS2rYmZWGuiS68SkcwIJQVB3jbtX5wFhFWf/purIs2W7Hzg9Tfup5AQQg7f/yCZJGOMLv+96OZb/AUuX0W/UdDRFsDeFHt0c+U2XHfp2YdsYSrtzRy9ZZGv0hXfOC6uDSf00c7xpRbSEdkwOEH//tlrHgqYiBoc9dHr6a6bvpLMMciDo7jkV8YRETY2bYDr7sDLjp1W2eUX756jqOnByjMt3nr+nLWLCvi9f09UxrKCQaE1UuLMAb6BlxWLy3mulVl5Icu9Lnq37qGYFE+b377GcKn2imoLePy37+F+i1rcaMxOvad5OxvD3DskVdSnqv19YO4UQc7NHv+vPMWxChaHhtVPC18IEi0Tcfvs2n2/EaoWWtklk3dkgo233kFLz9xcNxsntEvAm7MxRiGa/DHIi7//q1tfPjzN2Hb03Oz2dcT4YVH9nG6qQOA4vJ86rcUsX5lK5vbOhks3Qz4i62eee08T7/eMTxi1dvv8rMX21mzrJjCPIvwYPKf1xKwLMFxzfCItGVBMGDhuIYNK0p575Za7HFmfGs3LKd2w/Ixj9uhINXrGtn5lUcYt5SnAeO4MEsCvl3iUbQ8htijR+uLV8bo6rXxBnSgIVtmx2+EmlNWbVzIwuVV/PB/vZzWsMxQlkaiGOW6HqcOnWfJypoMt9J/7Z9943XCPYPDE4Y95wcI/3yAe6qjxGo2Dx/7L79sYdeR8Jjpiahj2HGolw+8rZYfPNuW9FyBgPCROxbQdKaf4oIAV19RgudBVzhGdVmIooKpZyid29FEtLt/3LrNxYtqsrvT1Tjy62OJ52UF8hY4DBxNr3a+mjoN+GpSSisKuPyqOo7sPjtqeMcOWBhjRu2VK5ZgRpb3HSEWcdn6/FFKqwqorM1sKYYTB9oZHIgNB/shnmt48rkabrlmkCPN/QxGPPY2hZPGUc/Aj55LHuwBasqDRB2PypIgddUhCvJsArY/Tp8p/W1d408oCKz97Lsyds5MsPJNwro6YoGdryuuskkDvpq0m+5ZSX5hkH2vNeM6HqGCAIUleXS0jt6M23NNysy7cy29PPLg67znk9dSNX/seH5fzyBvPHuU42+248Rc8vIDGCO4rsf8hnI23rqMsqpC7IA1qgZ9R2uYWCRBERwj7D8WZv+xMF68aeOlv493I3O6Pcq//LJl+HUsgZvWl3PXDTXjDuOkY7Cjl/yKYhLvbzWCCFVrGqd8vkyKdVgEy7wxNXWM4z+nskcDvpo0y7bYdPvlbLxtObGoQygU4DtffCFhT962xV9tmiiH34ATdfntLw5z98euHvXUuZZeHn3wtVG5/070QimHEwfaOXGgHYBQfoC1Nyzmmi1LEUsorSwgELITVv6cajG0REa+aXgGfrOri5hr+J0t8yb9mj3HW9n+339M+GTbhbGxFHvPz8YSBZEzQQoWOSAXyiUbD7yYEGnVEJRNerXVlFmWkJfvZ1uE8gNEBsYWF7Nsi8bV8zi840zSoZOzJzoB6A9HeP3pIzTtayOaYG/XZKKDDjt+fYzezgECQZvmox04qTYFn2auB6/t6+HO66spmODmJwDRnn5e/n++QaxvcHQgT3XDYAyx8ADB4inm9GeQcYTurfkULosRqnHBQKTNpv9oCDxN0cwmDfgqo67ctJg3nj0yNm3TwNHdLSnnG4OhAJGBGD/96mv0hyNjxt7T4TqGg9svLkBmmKnVnAFb6OhxqK+ZeMA/9dR23JibYIfoFFk6Am1bD1O/Ze3EGzuNvIhFeP/smUjOVTqApjJqzfWLWHxFDXbQwg5aBEM2obwA85eUp5xvtAMWqzYu5MAbzQz2j51onZqJBftQUJhfGSQYuPB9AVsIBWTC++U6rqF8khO33U0teImKqaWYUJjJcshq9tPfDpVRlm1x271r6WgL03Ksk/zCEA0rqvn5P23FJAlUIlDXWMHVWxp54rvbJ5bfPw0EuG1jFV1hh9f29+B5hquvKGXtsiK+9shpoo5HzDHx3HuorQjRcj46ptMdsIU1y4rTTsl0BqJEe/rJryrBCtiULJk3tCPL2IMDFiS4TsbziPUO0N/aSeG81JutqNyjAV9Ni8ra4lFpltV1JZw70zMmKFq2sGbzYq6/3a+bn1848ysvHdfQ0++y5epKtlw9ui7N5+9bwqt7u2k6M0BVaYAb1lZQVRbkuTfO8/z2DmKOH6NF4MqlRXzg1vEnbN1IjN0P/JzTv9qNiCABi8t//20s2LKGA//0yySN9AiWFOBGHf8uID7MI5bFvod+wd6vPcGi269h7Z+8a1TmksptGvBVVqy7sYHDu1pGj+2LP25/zZalww8Vlsz8OK9tCwuqE7ejKN/mbRsqedtFj79jUzXv2FRN36BLR3eMipIAxYXp/Xlt+/sf0rb1MF40PkEdgYPfeQY3GvMrrCXJGc2rKGbJu66j9bVDnNtxBOOYC68BND+znYorFrL49msSfr/KPVMawxeRShF5RkQOx/9NeA8pIsdFZI+I7BSRrVM5p5qbymuKuPMjV1FaWYAdsLBsoWZBCe/+5LWE8v3AaDxD64mulK9Tv6wy6ebYiQQDQjAg3HV9Fels/mRbUF0WYln95LJcivJtFs3LTzvY97d2jQ72cW4kxtF/eynlAoHB870sffdmFr5tXcKxe3cwxtGfvjSxH0Bd0qbaw/888Jwx5ksi8vn41/8lybE3G2POTfF8ag5b0FjJvf/xBvq6I1gBGd4EHaDleCfP/HAPAwm2Sxxy+VV13HD3FXz7v72Q9Ji8gAdis3F1KaGARWlRgKsuL6W4wObyhiJ+9us2jrUMJvxey4J1l5XwOzfXZm0YJNzcjhUMjAn4AE5v6m0Uh8ogR7v6/No5CUS7+6beSHXJmGrAvwfYEv/8u8ALJA/4SiEiFJeP3tg83D3IE9/dkXCB1BA7YLH+LUsIpSgIFgwZ3r6xlpvWlhMIjL0NWFSbzx+/fzEt5yN898kztHWOzoCxLWFlY9GkcuYnIxYe4PTzu3H6Er8BpWQJV9znDyxVXtmA2BZcvObAEqrWzq5Vt2pmTTUtc54xpgUg/m9tkuMM8LSIbBOR+1O9oIjcLyJbRWRrZ+f5KTZPzQUH3jideAUugPjB/vo7llM5rxixhLxkE7uexcqGooTBfqSyogAdPWN71DHH8O8vnfPr/kwzNxrjxc88SPNzOyf1/bUbljN/80oAyq9YSOXqBqyRb4YCdl5w+E1BKUijhy8izwLzEzz1hQmc5wZjzBkRqQWeEZE3jTEvJjrQGPMQ8BDAytXrZuFC8UuPNxjl/MOP0f3IC5iYQ/Fbr6L6j95HcH5VVs7fda4PL8lYdXlVIe/6xIZRk7nrbmhg+wtNo4u2WYb5lXnUVY0/6Xu2I0rA9ksZXyw84DAY9aa9l3/il9sYaO9OOhSTsnwCMP+G1UR7BwiVFCAibPxv93H4By9w4onXcfqjVK1rZNUf3E7J4mR9MJWLxg34xphbkz0nIq0iUmeMaRGROiBhSUFjzJn4v20i8iiwEUgY8FV2Gc/j5Mf/nsjBE5j4Ip/ux18i/OsdND7yZQLVZdPehpqFpRx/sx33otW5li00rqodk7lz1U1LONHWQtuePgJBAw7U1xTwsbsv2n08iZJCO+m+vJYlhILTtx6x/2wn27/8Ezr2nkixWjZ1vXsJ2Oz/xpPs/erjXHHfLSz/4BbsUIAVH7mVFR9J+ueq1JTH8B8DPgJ8Kf7vzy8+QESKAMsY0xv//Dbgb6d4XpUhfS/vJnLk1HCwB8D1cMMDdPzzL6j90w9OextWXF3PjheO+QuuRsQ5O2CxetOiMcfvOX+CJRtjfO7aHk50rqK0KEBNefo11WvKQyyoyeNU6+CoNU1BW9i4qjQj1S2HuFGHjj3HMRjKL6/nN3/yIJGuvnE2MEl9Y2scFyd+Z3DoX35FaWMd8667ImNtVpeuqQb8LwE/FpGPAyeB9wOIyALgm8aYO4F5wKPxrIcA8H1jTJLVJCrb+t/Yj+mPjH0i5tD30k7IQsDPLwzynk9t5IWf7qOtuQeAqvnFvPW9qykuyx9zvGc6uaOhl+DOYpbVF07qnP/h7gV842enOdcVHd6p6vLFhbzrxsxtxNLy8n52fPkngJ9y6kZjfjBPFc8tGf+YEdzBGEd/8hsN+CotUwr4xpjzMGYNytAQzp3xz5uAdVM5j5o+dnmJvxVegrRAu6I0a+0ory7i3Z/cSDTiYIwZrr55sZ2nX6Wxsouqfb3EFmxOeEw6SgoDfO7exZxuj9DRG6OuKm9CdwnjCTefY/sXf4SbqBZOIuLXwVl8+wZOPbUt/e8DBtq7J9lKlWt0pW2OK73rBs597ZGxBRkL8qj4vXdkvT2hvADOuS7O/fOviZ1sJX/dZZTduRmrMJ+mrg5MLMLmo52jtiecLBFhYW0+C2vH3kVM1fHHX8ObQNH9suX1XPf395FfUUKotIDDP3oRk05pZ0uovHLJ5BuqcopWy8xxwXmV1H3xU0h+CCnMR/LzkLwg5R+4leItV4//AhnWv/UAR+/8j5z/xqN0/+zXtP2Pf+Ho3Z9j595f09W+gzsaejMS7Kdb/9lOzHjbaA2xBKdvkMP/+iv6WztZ8dG3c/M/fZay5Qv8/PoU7Lwgyz+0ZeoNVjlBspFzPFkrV68z3/rxUzPdjJzg9vYTfmE7JhKlaPMaggtGj2W7fQN0//xF+l/fT2B+FeXvvwWvs5e+1/djlxRSesf1BKrLxz2Pv7etX+RrzHOux5Fb/gi3o2f045bgXlXPB/7T+lkR7HuOt9J96DT5NWVUr2tM+LMc/elLHPjmU3gT2IBFAhZ2KMiND3yK0iXzcKMxdn7lUVpe3IsVsnEjDoXzyhk814MXc6hc28iVn76LsqV1mfzx1BxX/bZN24wxGxI9pwFfjcs518XxD/wFbm8/ZiDiF5wxxv/XcZFgEATqvviHlN52XeLXON9N63//Lr3PbQXPo+CqK5j35x8h//LFw8cM7DrMyU9+CZNg5anYFnc98TdY6RTEmSZu1OGNv/5Xzu1qQixBEIIlBWz+h49TWFfJ6ed30fTob4mFB6i5ahknn92BN5C8VESy0sfV65ey+R8+Mfx1pLuPgbOdFM6vJFQ2uUlqlTtSBXwdw1fjavuH7+N09FzYCHZoqMLzvzZRf4LxzH/5Ku3/5yeYwSjFN62n6v53E5xXiReNcez9f4Y7YnJxYNubnLzvb2h85EvDdxPGcREkcYKKMUnr6WfLgYef4tzOo6Pq3jiDUV79s+9QuWYJZ17YjTvoX4u+lvME8kL+Dn6Jmi0QyAviJHhDOLfrGMaY4Xo+eWVF5JUVTcePpHKMjuGrcfU+vzW9Xb8dl9jxFpyz5+l65AWOv+/PiLV10v6PPxoV7Id4/RHOf+fJ4a/z1yxL+tIVKxdjp6ijM92MMRx/4vWxRc6MYaC9h+ZndwwHewBcg9MfIVhcgFx0VyK2xbLfuZFk22dZ45SGUGqy9DdLjW8yw36Oixse4NxDP6PzJ88lfd3+bW8Of2mFgsz7648j+SFMfPGTBCwChXms/ew9k2k5AJ7jEgsPpl0jx43E6HyzmfDpC7WcnP4I3mDiVEnjuUknaN1IjJqrlmEFbez8EIGifK789N2s/MQ7Ek7ISsCm7qY1ummJmhY6pKPGVXzzNfQ+/VrK2uwJOS7hZ9+AFOPYXriflr95mKKNqyi59VqOXVOO+bs7qHhuNwWHw1RedQVL33sDBTXplXgYaO8GY8ivKcOLOux98AlOPbMd4xryyotY/ck7qL85+bKQoz99mTe//TRiWXiuS/HCGjb+7e/TtvVw0vo2JsWWjF7U4arPvx9BiPb2Uzi/Ynge4tq//j1e+8L3MMbDizjYBSHyKoq58tN3pfWzKjVROmmrxhVr6+T4B76AFx7ADEb9wvGel3y/1ZGS7L06hiV45QW4K+dR3XKe4oYlXPa7N1G2PHF9HON6ND+/ixO/eAMTc6la00jLbw8w0NoJQEFNGaGyQrqPtIwahrHzglz95x+g7oZVY17zzG/2suNLPxm96MkSCqrLqFq3hOZnklS2TLErlRUKsPkfPkHlqsUJn49099H87A4GWruoWLWYuhtW6Ubkakp00lZNSbC2gqWP/QNdj7xA/2t7Cc6vouQdmwi/uIPwizuxCvOJHDqZeJw/3Q3JPYN09BN8+RjdQPexPZx95QDX/MUHmX/9ylGHGs/j9b/6Z9p3HsWL+MG888CpUcf0nT5P3+mx5bXdSIwDDz+VMOAf+pdfjV3h6hmivf1g/OGWhNUtU9z5GM9QWJs8XTWvrMgfz1cqCzTgq7TYJYVUfeROqj5y5/BjRdetZt7/+/sAnPv245z7Xz+c0jlGjVobgxuJsfMrj/COH//ZqFz3lpf30/rGofHvLpJI9EYAMNCWeHtF43qULK71i1hO8FyVVy4mvzp7JSqUSkUnbVVGmMFY0qyTqXAHY4RPtY96bM8Dj0062IO/+XciJQ2Ja8eLZVG5poGKlWMrd6YksOEL9060eUpNGw34KiOs/BBJdwkX8cf9AWwLyUu/SJkxHlbwQiG1rkOnifb0T7qddl6Qyz5wU8LnVnz07dh5o4u2ScCmaGEVlVcuIVgysUVPNdcsT/rmotRM0ICvMqLktuuQBHXkJT+PyvvfTeldm8lb1UjZe7bQ+G9fhPrq8YdHBArnV1K0oHL4oZ7jrYid3mpbOy9I6bL5WHlBAkV5WKEAje++nsZ3X5/w+Or1S7n6zz9AQW0ZVsDGCtjM37ySzf/zE36htbetw85P/82qdFmijeKUmjk6hq8yIlRfQ+3nPkTbV77v56S7LpKfR+GGldR86j2jFh81dXUw+OV3UvrZHxDrimJcDwlYGNdghQJ4kRh2fhArEGDDX1yoxx/t6ef8zia82Pilg/NrSrnmC/dSdWUD4dPniXT0Uto4j2BxQcrvq7thFfM3ryTa008gPzSqx193w2pOXLmVjr3HRy+ySsAuCFF+Wf247VQqmzTgq4ypuPc2im5cR88TL+P2DVJy01UUbFgxZhFRT6yVxssdNv3jLTQfrSR8sp2i+iqqr1rG2Zf20X3sLCWLa1h4y3oChf72htHufl745AP+cE6yWwMBKxDgsg/exBX3vW34vMX1VRTXp78/r4gkLGUgtsWmv/8IZ17cy6lndyAi5FeXcOKJN0a3ybYIlRRSd+PYTCClZpLm4aus2tm2A6+7g4+WdDIo16b9ffu/+UuO/vTlhDXiJWBTsXIRNVcto/5t6ycU3DOhY/9Jdv/jz+g97m/pXHvt5az703eTX6XZOSr7NA9fzSoTDfYALb/Zl3hDEEtY/qEtrLhvzMZrWVO5ajFbvvEnOAMRxLZntOaPUqnob6bKit3tx3G6jrJsYQz3aAwmuHXsxdkzQ6yATag48ztWTUagIG+mm6BUSlPK0hGR94vIPhHxRCThLUT8uNtF5KCIHBGRz0/lnGru2Xn6VZyuo9zXEObanpWT2sSk4e6NSYN+3U1XTrWJSuWEqaZl7gXeC7yY7AARsYGvAncAq4B7RURns3LE7vbjLKvr596wQ7Rz/aRfp+Gua6la2zicFikBGysUYM1n3klBdXqF1ZTKdVMa0jHGHADGK+W6EThijGmKH/tD4B5g/1TOrWa/3e3HcftOsLy6H0jcO0+XZdtc98WP0LHnOK1vHCJQmMfCm9dROL8iM41VKgdkYwy/HhhZ2aoZSLwPnrpkNHV14EXP8uEFXUTb1mMSF72cEBGham0jVWsbp/5iSuWgcQO+iDwLJFoy+AVjzM/TOEei7n/SXFARuR+4H2BenS5cmat6Yq00VnZhDvVPeIJWKTU9xg34xphbp3iOZmBk1amFwJkU53sIeAj8PPwpnltlWVNXB91dezAxh7e4YQYnMUGrlJoe2RjSeQNYLiKNwGngg8CHsnBelWU72w/gdZ3hzssGKG1bw+BMN0gpNcpU0zLfIyLNwPXAEyLyVPzxBSLyJIAxxgE+AzwFHAB+bIzZN7Vmq9nqvoYwpW1rZroZSqkEppql8yjwaILHzwB3jvj6SeDJqZxLzW47T7+K19+H5cYSz9oopWaclkdWUzYy136iJROUUtmjAV9NyXCuvUx+UxKlVHZoLR01adORa6+Umj7aw1eTNirXXik162kPX02Y5torNTdpwFcTorn2Ss1dGvDVhN3XECbaNvnKl0qpmaEBX6WlqauDnr5Dmmuv1BymAV+lJRzr4a7L+ih+xWGwVHPtlZqLNEtHjWtoe0K3f2Cmm6KUmgLt4auUhkomDI3ba669UnOXBnyV1M62HTRWdrGxTaa0PaFSanbQIR2V0O7245j+Ht7SHcYsWDHTzVFKZYAGfDXGzvYDOF1H+fCCLi2GptQlRAO+Sui+hrAO4yh1idExfDVsZMkEzbVX6tKjAV8BWjJBqVygQzqKpq4OiHbz0ZJO3Z5QqUuYBnxFT98hGiu7cM/FZropSqlppAE/xw3n2r8pxLTMsVKXtCkFfBF5v4jsExFPRDakOO64iOwRkZ0isnUq51SZo7n2SuWWqU7a7gXeC3wjjWNvNsacm+L5VIbsbNuB193BfQ1hBjs1116pXDClgG+MOQAgovl7c8XQpuNL5w2wua2TaKcO4yiVK7KVlmmAp0XEAN8wxjyU7EARuR+4H2BeXX2Wmpdb7l7pUHR8JbGamW6JUiqbxg34IvIsMD/BU18wxvw8zfPcYIw5IyK1wDMi8qYx5sVEB8bfDB4CWLl6nUnz9VUahoZx3Gotc6xULho34Btjbp3qSYwxZ+L/tonIo8BGIGHAV9OjqasDYhE+HGkj1qbDOErlomlPyxSRIhEpGfocuA1/sldlydD2hI2VXTPdFKXUDJpqWuZ7RKQZuB54QkSeij++QESejB82D3hJRHYBrwNPGGN+OZXzqvTtbD9AV/sObl9wTnPtlcpxU83SeRR4NMHjZ4A74583Aeumch41OUP1cT5a0slg27W6W5VSOU5X2l7Kot1+nr3WtFdKodUyL0kjc+1j+/JBe/ZKKTTgX7LuaOiltG2NDuMopYbpkM4lZmfbDpyuozPdDKXULKQ9/EvE0DCOiTmaa6+USkgD/iXCM518eEEX0c71xLhsppujlJqFNOBfAna2H8Dr7sDydB9apVRyOoY/x43Ktdf0S6VUChrw5zrNtVdKpUmHdOaooZ79soUxzbVXSqVFA/4cM1QIzevv84dxerRkglIqPTqkMwddtsTw4UibDuMopSZEe/hzyO724zhdR6nKi+HEKme6OUqpOUYD/hzimU7uawgT7VyvwzhKqQnTgD9HDG1PqLn2SqnJ0jH8OWBoYZWO2yulpkJ7+LNcU1fHcK59tFPr4yilJk8D/iw2VBDtjoZeYjs1114pNTVijJnpNiQlIu3AiRk6fTVwbobOPRlzqb1zqa2g7Z1u2t7MajDG1CR6YlYH/JkkIluNMRtmuh3pmkvtnUttBW3vdNP2Zo9O2iqlVI7QgK+UUjlCA35yD810AyZoLrV3LrUVtL3TTdubJTqGr5RSOUJ7+EoplSM04CulVI7QgB8nIu8XkX0i4olI0pQrETkuIntEZKeIbM1mG0e0Id223i4iB0XkiIh8PpttvKgdlSLyjIgcjv9bkeS4Gb22410v8T0Qf363iFyd7TZe1J7x2rtFRLrjBCsVJwAAAtZJREFU13OniPzVTLQz3pZviUibiOxN8vxsu7bjtXfWXNsJMcbohz+PsRK4AngB2JDiuONA9WxvK2ADR4GlQAjYBayaofb+D+Dz8c8/D3x5tl3bdK4XcCfwC/zydZuA12bwdyCd9m4B/n2m2nhRW24Crgb2Jnl+1lzbNNs7a67tRD60hx9njDlgjDk40+1IR5pt3QgcMcY0GWOiwA+Be6a/dQndA3w3/vl3gXfPUDtSSed63QN8z/heBcpFpC7bDY2bTf+/4zLGvAh0pDhkNl3bdNo7J2nAnzgDPC0i20Tk/pluTAr1wKkRXzfHH5sJ84wxLQDxf2uTHDeT1zad6zWbrmm6bbleRHaJyC9EZHV2mjYps+napmuuXNthOVU8TUSeBeYneOoLxpifp/kyNxhjzohILfCMiLwZ7w1kVAbamqhq/rTl4KZq7wReJivXNol0rldWr+k40mnLdvy6KmERuRP4GbB82ls2ObPp2qZjLl3bYTkV8I0xt2bgNc7E/20TkUfxb60zHpQy0NZmYNGIrxcCZ6b4mkmlaq+ItIpInTGmJX6b3pbkNbJybZNI53pl9ZqOY9y2GGN6Rnz+pIh8TUSqjTGzsfDXbLq245pj13aYDulMgIgUiUjJ0OfAbUDCWfxZ4A1guYg0ikgI+CDw2Ay15THgI/HPPwKMuUOZBdc2nev1GHBfPKNkE9A9NFQ1A8Ztr4jMFxGJf74R/+/9fNZbmp7ZdG3HNceu7QUzPWs8Wz6A9+D3MiJAK/BU/PEFwJPxz5fiZ0PsAvbhD6/MyrbGv74TOISfzTEjbY23owp4Djgc/7dyNl7bRNcL+BTwqfjnAnw1/vweUmRzzZL2fiZ+LXcBrwKbZ7CtPwBagFj8d/fjs/zajtfeWXNtJ/KhpRWUUipH6JCOUkrlCA34SimVIzTgK6VUjtCAr5RSOUIDvlJK5QgN+EoplSM04CulVI74/wEsEYnRC4FzvQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the resulting classifier\n",
    "h = 0.02\n",
    "x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
    "y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
    "                     np.arange(y_min, y_max, h))\n",
    "\n",
    "Z = np.dot(np.c_[np.ones(xx.size),xx.ravel(), yy.ravel()], w) \n",
    "Z = np.argmax(Z, axis=1)\n",
    "Z = Z.reshape(xx.shape)\n",
    "fig = plt.figure()\n",
    "plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral, alpha=0.3)\n",
    "plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.Spectral)\n",
    "plt.xlim(xx.min(), xx.max())\n",
    "plt.ylim(yy.min(), yy.max())\n",
    "#fig.savefig('spiral_linear.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见softmax回归本质上仍然是线性函数模型，表现在图形上其分割线都是一些直线，很难对数据进行非线性分割，模型的准确度是 0.5366666666666666 。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.7 批梯度下降法和随机梯度下降法\n",
    "\n",
    "下面代码读取MNIST手写数字训练集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50000, 784) (50000,)\n",
      "(10000, 784) (10000,)\n",
      "(10000, 784) (10000,)\n",
      "float32 int64\n",
      "0.98828125 4\n",
      "0 9\n"
     ]
    }
   ],
   "source": [
    "import pickle, gzip, urllib.request, json\n",
    "import numpy as np\n",
    "import os.path\n",
    "\n",
    "if not os.path.isfile(\"mnist.pkl.gz\"):\n",
    "    # Load the dataset\n",
    "    urllib.request.urlretrieve(\"http://deeplearning.net/data/mnist/mnist.pkl.gz\", \"mnist.pkl.gz\")\n",
    "    \n",
    "with gzip.open('mnist.pkl.gz', 'rb') as f:\n",
    "    train_set, valid_set, test_set = pickle.load(f, encoding='latin1')\n",
    "\n",
    "train_X, train_y = train_set\n",
    "valid_X, valid_y = valid_set\n",
    "test_X, test_y = valid_set\n",
    "print(train_X.shape,train_y.shape)\n",
    "print(valid_X.shape,valid_y.shape)\n",
    "print(test_X.shape,test_y.shape)\n",
    "print(train_X.dtype,train_y.dtype)\n",
    "print(train_X[9][300],train_y[9])\n",
    "print(np.min(train_y),np.max(train_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可视化其中的一幅图像："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAADtCAYAAABedtXhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAc8UlEQVR4nO3dfZBV5Z0n8O+XFkUUVMQwBPFlApkpysxihqDRjRqdZNDaDXGyjpAxo+YF0WAlKdnC6NY6s+SFzIw6xjHGji+RjTsW5eLAWEwY4jhRN2IACwVk1dYYbGHEFl0ZTWl339/+cc8l57nn0vfce59z7nNOfz9Vt+jn9HPveRp+58fTz3nO89DMICIi5Tam2w0QEZHsKdmLiIwCSvYiIqOAkr2IyCigZC8iMgoc0u0GyOjyx588wt7YN5yq7pZn3ltvZvMybpKIF/PmzbOBgYFUdbds2ZJ7bCvZS67e2DeMX64/IVXdnqkvTM64OSLeDAwMYPPmzanqksw9tpXsJVcGoIJKt5shkomQn1tSspdcGQyDlm4YR6RoKpVwOzJK9pI79eyljMxMPXuRGoNhOOALQqQTSvYiMRWEe0GIdELJXiRiAIaV7KWklOxFYtSzl7JSsheJGIDBgC8IkXaZmWbjiNQYTMM4Ulrq2YvUGDAc7vUg0hEle5FI9QlakXJSshc5gBgGu90IEe/0UJVITPUGrZK9lJNu0IpEqvPsleylnNSzF4mpqGcvJaRhHJEY9eylzJTsRSIGYli7YUpJKdmLxGgYR8pKyV4kYiDet55uN0PEOy2XIBJTfahKwzhSTurZi8ToBq2UlZK9SMSMGDb17KWclOxFYirq2UtJKdmLRKo3aBV2Uj66QSsSoxu0Umbq2YvEDGuevZSUkr1IRE/QSpkp2YvEVDQbR0pIC6GJxFQXQlOyl3JSsheJGIhBLZcgJaXZOCIRM+ihKikt9exFDqAeqpJS0pi9SIxBPXspLyV7kRjdoJWyUrIXiRiozUuktJTsRSIGYFBr40gJaW0cEQe1nr2Ulnr2IhGDnqCV8go52euqk9wNR737Zq80SM4j+RzJPpLXNvj+UST/keTTJHeQvNz7DyQSqU2/bPZqJou4Vs9ecmVGbz17kj0AbgPwKQD9ADaRXGtmz8aqfRXAs2b2n0keB+A5kveZ2fteGiES46Nnn1VcK9lLrqo3aL0tlzAXQJ+ZvQQAJO8HMB9A/KIwABNIEsCRAPYBGPLVAJEajzdoM4lrJXvJWUt70E4muTlW7jWz3lh5GoBXYuV+AKfVfcbfAVgLYDeACQAuNrNwp0xIobXQsx8ptjOJayV7yVX1Bm3q2TgDZjZnhO83+qD6q+2PAWwFcC6ADwHYQPIxM3s7bSNE0moh2Y8U25nEtW7QSu6GMSbVK4V+ANNj5eNR7enEXQ5gtVX1AfgVgN/38oOI1PF0gzaTuFayl1zVnqBN80phE4CZJE8meSiABaj+ahu3C8B5AEByCoDfA/CSxx9JBED6RJ8i2WcS1x0l+2bTg0QaqWBMqlczZjYEYAmA9QB2AlhlZjtILia5OKq2HMAZJLcBeBjAMjMbaPbZim1ph49kn1Vctz1mn3J6kONQHmbjcES7p5SC2I83B8zsuEbfMwMGK/5+oTSzdQDW1R37Yezr3QA+3cpnthPbJMN9mka8MRv5V05fyyVkEded3KBNMz3IMQ5H4DSe18EppQh+Zg/8+mDfqw7jBD962HJsiwBhP0HbSbJPMz0IJBcBWAQA4zC+g9NJWRRgbZyWY1ukzJuXpJkehGjuaC8ATOSkcP8mJBctTr3slpZjW8M4ApS3Z59mepBInUIM4yi2pS0hJ/tOrro004NEEirRPrTNXl2k2Ja2+FoILQtt9+zNbIhkbXpQD4C7zWyHt5ZJKVVn43hbGycTim1pR6k3L2k0PUhkJEXZllCxLe0IeRhHa+NI7ro8RCOSGSV7kUhBZuOItEXJXiSmALNxRNqiZC8SMSOGlOylhEp9g1akHRrGkbJSz14kojF7KTMle5EYJXspKyV7kUhR5tmLtKrMC6GJtEXz7KWslOwlCMf+n2Oc8pgGCzW+fsZbmbbBDBjyuHmJCAA8/PDDTplMdijOPffczNuh2TgiMRrGkbJSz14kojF7KSuN2YvUabKNp0hhlTbZk3wZwH4AwwCGzGyOj0ZJuRXhBq1iW9pR2mQf+aSZDXj4HPHo+buSuWnTCbc45Y8/9tVEnd/F1szaBFRv0BZoGEexHaCbb745ceyMM85wyitXrsyrOY6yJ3uRFhDDmo0jJRT62jidXnUG4J9JbiG5qFEFkotIbia5eRDvdXg6KQMzpnp1u5loIbZzbpsEqpTbEkbONLPdJD8AYAPJ/2tmj8YrmFkvgF4AmMhJ4f6OI7ko0No4LcU22eChBRl1SjuMY2a7oz/3knwQwFwAj478LsnC87fPdcqbPp0c19xfcQNx4s8Pz7RNDVl13D50iu1wrFixwikvXrw4UWdwcNAp1z9klZeQk33bwzgkjyA5ofY1gE8D2O6rYVJeFTDVq1sU29Kusg7jTAHwYPRY8iEA/peZ/dRLq6S0rBg3aBXb0rLQb9C2nezN7CUA/8FjW2SUCPg3XQCKbWlfyMM4mnpZEuecutMpTxhzaKLOVb+e55Qn3/FEpm06mABm2kiBnH766U557NixiTqPP/64U161alWmbToYJXuRiJmSvZSXkr1ITEGmXoq0RAuhidQJ+HoQ6YiSvUjEQFTCn40j0pZSzsYR4Dfz5yaOTb7mV075vYt7nPLQnn/zcu69V7kLP31vivsQ1U/ePjHxnje/eYJTHoM3vLSlVeH2faTmrLPOShy7/vrrnfLChQud8r59+7ycu/5zTznlFKf84osvJt6zdOlSL+fuROjDOOpiSb7M79o4JOeRfI5kH8lrD1LnHJJbSe4g+XOvP49IjK+HqrKIa/XsJX+eOj8kewDcBuBTAPoBbCK51syejdU5GsAPAMwzs13RWjcimfDRs88qrtWzl9x57NnPBdBnZi+Z2fsA7gcwv67O5wGsNrNd1XPbXq8/jEiMp559JnGtnn0HLlnxUOLY5RNfccp/9IdXOuVxD/kZs7/0q+uc8uzDDnPKX1l+YeI9kx7rzkNUcQagUkk99XJy3fLBvdFKkzXTAMT/wvsBnFb3GR8GMJbkvwKYAOAWM+vOzhYF0tvbmzg2c+ZMpzxr1iynXP9gU7uuu+46p3zsscc65a985SuJ9zz99NNezt2JFpdLGCm2M4lrJXvJlwFIP89+oMl2gI0+qL7bdAiAPwRwHoDDATxBcqOZPZ+2ESJptTCMM1JsZxLXSvaSO48TFvoBTI+Vjwewu0GdATN7B8A7JB9Fdd0bJXvxztNsnEziWmP2kj9L+WpuE4CZJE8meSiABQDW1tVZA+ATJA8hOR7VX4d3QiQDnsbsM4nrpsme5N0k95LcHjs2ieQGki9Efx7T7HNEqtLdnE1zg9bMhgAsAbAe1UBfZWY7SC4muTiqsxPATwE8A+CXAO40s+2AYlv885HsO43rg0kzjPNjAH8HID74fy2Ah81sRTQH9FoAy1J8Vqnsef/oxLEKfu2Uhw7vfB2YytmnJo7NP/JWpzxo7q5TQ+MCXn/G43MnZrYOwLq6Yz+sK/81gL9u8PYfQ7Hd0Lvvvps4Vp+kxo0b1/F5Zs+enTh24onuA4H1Nz19nDcLPh+q6jCuG2ras4/23ax/NG4+gHujr+8F8Nm0J5RRzgCrMNUr86YotsWzSqWS6tUN7d6gnWJmewDAzPaMNKGf5CIAiwBgHMa3eTopl4B/62gztkWAUb4QWjR3tBcAJnJSuH8Tkp+SREE8tkmW5KeSTpQx2b9GcmrU85kKYFQ8lfjC993nGh489tZEndvf+rBTPnrjq055KMV5eo4+yikPLH0nUeeDh7gPUX1jt7sw2pS7tiTeE0wYBtOQhkZlbC9fvtwpf+QjH0nU2bnTnezRzoNMRxxxhFNetix5O2T8eHcEYOPGjU75gQceaPm8eSjrQmhrAVwafX0pqtOARJqrPVSV5tUdim1pm6+F0LLQtGdP8u8BnIPq4739AG4AsALAKpJfArALwEVZNlLKJZTOj2JbfAu5Z9802ZvZwoN86zzPbZHRIoeZNmkotsU3bV5SUD2/N8Mp/8//dLtTftcGE+9Zff2nnfLhr/yy5fO+8IOTnfL2j/4oUednv5ngvudj77V8nm7Rrczumz59ulOuX1xsaCh5d2nJkiVO+fXXX2/5vDfddJNTvuii5C9Ou3e7KwOceeaZLZ+nG0Ifs1eyl3ylXwpBpHCU7EUO6OrNV5FMKdmLxIV7PYh0RMleJC7ce1gibWtx85LcKdlH7MzkgkwL7nJ3oppz2LBT/v2ffi3xng//Q+s3ZF/+1sed8uazbqqrkfxnWnbnF53yNPyi5fN2RWubl4gHp5xySuLYgw8+6JQnT57slG+9NfnA4M9/3vpe7UuXLnXKl112WdP3fPvb3275PKFQz14kRrNxpKyU7EXiwr0eRDqiZC8iMgoo2XcZxx6aOLZnibvX7+alyTHKsexxyoPmLiX0J7OfSrxn7ffc8fcZf+kuFjXmd5Ir5n7mAnehp566JYBn/8IdnweAE1YUZIy+AQ3j+HPIIclL+JJLLnHKd911V6LOmDFuLNffWPz4x904BoBvfvObTrn+AalJkyYl3lP/0BTpxvbKlStR74477kgcKwI9VCUSZwhmuQQR3zQbRyQu3M6PSEdC7tm3u+H4X5B8leTW6HVBts2UMqGle2XeDsW2eFboJY7ReFNmALjZzP7Ge4sy8G+L5ySO/XLpLU650S9fg3X/JivfnuaUv/M7Tybe851L3GPX/ZG74cmnjvqnxHs+efi/O+Un33M3VD7hom0NWldg4XR+foyCx/aCBQsSx+68806n3Ci51A839PX1OeU5c5LXTP2x+fPnO+Vp09zrAwCmTp3qlOsXT/viF5P3o4qq8GP2ZvYoyZOyb4qMGoFcD4pt8S3kZN/uTlUAsITkM9Gvwsd4a5GUWtohnC7P2FFsS1tCHsZpN9nfDuBDAGYD2APgxoNVJLmI5GaSmwdRnDXXJUMVpnt1R1uxnVfjJGyVSiXVqxvamo1jZq/Vvib5IwAPjVC3F0AvAEzkpHB/x5HchDzPvt3YJkP+qSQPhR+zb4TkVDPbExUvBLB9pPp5e32x+0DIL5b9baLO/oq7y9Szg0ck6ly/9AqnPO6N953yw995OfGee076Z6dcfxN3TINfpur/n59zqHueb/TtTLznls/9ifsZTyfrBCvc6yH42L744oud8j333JOoMzjoxvZbb72VqPP5z3/eKb/55ptO+cYbk7/QnH322U65/oZt/QNTQHIMu37BtVdeeSXxnnPOOccpv/jii4k6oSp0sj/IpsznkJyN6mX7MoArDvoBInHdH48/QLEtvhU62R9kU+bk89ciaQVyPSi2xbdCJ3sR3xjuE+UibdPmJV0w68/d8eu170xJ1PlOr9upm3pjcmGx8Ug+NBX3xjV/kDj2jVs/4ZRv/uBjI35GIz11Y5//ddvnEnU++PSzLX+uFN8VV7ijSrt27UrU+da3vuWUG43rN3P11VcnjtUvUNZosbRm6sf1H3nkkUSdIo3R11PPXiQu3OtBpCNK9iI1Ad2gFfFNyV4kLtzrQaQjSvY527J+llPed//kRJ2pz3W++cdvpoxLHLv6uH+pOzLWKZ3+P5Yk3jP56XdGPM/0vlcTx4Yb1CuMcK+H4K1Zs8Ypr169OlGn0dz1VtXPhwcab1wet3BhcnLT9u0jP6bQ39/fWsMCFvpDVZ2sjSPSMqI6GyfNK9XnkfNIPkeyj+S1I9T7GMlhkv/F048ikuBruYQs4lrJXvLlcSE0kj0AbgNwPoBZABaSnHWQet8DsN7vDyPi8rEQWlZxrWQv+bOUr+bmAugzs5fM7H0A9wOY36De1QD+N4C9nTZdZCSeVr3MJK6V7CV/6ZP95NqqktFrUd0nTQMQH6Duj44dQHIaqmvc/ND/DyLyW2kTfZTsR4rtTOK6lDdoT/hL9+arr5uZPccd55T7PzeUqDNj7GFO+b797k49k+94ouXzFvpmbAMtTL0cMLPklkmxj2pwrP7T/xbAMjMbbrRQV9HccsstzSu14aijjnLKF110UaLOxIkTnXL9w0+rVq3y37CCaeEG7UixnUlclzLZS+D8TVjoBzA9Vj4ewO66OnMA3B9dEJMBXEByyMz+wVsrRCKeZuNkEtdK9pIv87o2ziYAM0meDOBVAAsAOGv3mtnJta9J/hjAQ0r0khVPa+NkEtdNx+xJTif5CMmdJHeQ/Fp0fBLJDSRfiP7U9m2SjqcbtGY2BGAJqrMRdgJYZWY7SC4mubjZ+xXb4lOLY/YjfU5HcX0waXr2QwCuMbOnSE4AsIXkBgCXAXjYzFZE80CvBbCs3YYUwQvXzHDKO8/7fqLOE++5D1Gt+swn6moUd5EnX3wul2Bm6wCsqzvW8KaVmV1Wd0ixHbnqqquc8pVXXpmos3evO+nj3HPPzbRNReTroaoO47qhpj17M9tjZk9FX+9H9X+aaahOBbo3qnYvgM+mOaGIx6mXnTVDsS2ehbzheEtj9iRPAnAqgCcBTKlt32Zme0h+4CDvWQRgEQCMw/hO2iplkFMib1WnsS0ClGRtHJJHojqB/+tm9nba6T7acFziiPBWvfQR29pwXEqxeQnJsaheDPeZWW3lpddqmzOTnIoSPp3YM+vDTnn5hfc75eEG/4tfvta9fzLj+Y3+G1ZwIaXF0RrbJ554olP+8pe/7JQb9VB7e3udcpkWMfMl5J59mtk4RHVfzp1mdlPsW2sBXBp9fSmANfXvFWkokDF7xbb4VvQx+zMBfAHANpJbo2PXAVgBYBXJLwHYBSD5yJ1II+F0fhTb4lXIPfumyd7MHkfjx3cB4Dy/zZHSC2inKsW2+FboZC/iXbjXg0jbQt+8RMl+BH+6+l+d8oVHuvfpPrrx8sR7ZnxdN2Sb8bhcgrRpw4YNTrn+hu1PfvKTxHtuuOGGTNtUBoWfjSPiUyjDOCK+qWcvUhPoQ1UiPijZi8SFez2ItE1j9gX27TWfc8oLL3EXPjt8nbuZgzQX4hO0o9E999zjlJcvX+6U16zRowXtULIXiWEl3AtCpBO6QStSozF7KSkN44jU0TCOlJWSvUhcuNeDSEeU7Avqd5c94ZQ/s+xjTvlYuN+XdNSz777vfve7I5alPUr2InHhXg8iHVGyF6kxLZcg5RT65iVp1rOfTvIRkjtJ7iD5tej4X5B8leTW6HVB9s2VoqvNs0/zyrwtim3xrOjr2Q8BuMbMniI5AcAWkrVVlG42s7/JrnlSSuH8qqvYFq8KPYwTbbxc23x5P8mdAKZl3TApr1Bu0Cq2xbeQk33TYZw4kicBOBXAk9GhJSSfIXk3yWMO8p5FJDeT3DyI9zpqrJRA2i0Jc75mOo3tnJopAUs7hNOt/xBSJ3uSR6K6MfPXzextALcD+BCA2aj2jm5s9D4z6zWzOWY2ZywO89BkKTpW0r1ya4+H2M6tsRK0kJN9qtk4JMeiejHcZ2arAcDMXot9/0cAHsqkhVI6Ic3GUWyLT0WfjUMAdwHYaWY3xY5PjVW7EMB2/82T0jFUb9CmeWVMsS2+Fb1nfyaALwDYRnJrdOw6AAtJzkb18n0ZwBWZtFBKJ5QbtFBsi0eFXwjNzB5HdXp0vXX+myOjQiDXg2JbfCt0shfxSZuXSJkp2YvUmGnzEimtkG/QKtlL/pTrpYQKP2Yv4puGcaSslOxFagyAhnGkpJTsI/vx5sDP7IFfA5gMYCDPc3eoSO0Noa0njvjdcK+HTgwAKFpsF6mtQPfbO3JcQ8n+ADM7DgBIbi7SI+ZFam8R2upzGIfkPAC3AOgBcKeZraj7/p8BWBYV/x3AlWb2tL8WVBUxtovUVqAY7fWV7LOIaw3jSO58zcYh2QPgNgCfAtAPYBPJtWb2bKzarwCcbWZvkjwfQC+A07w0QCTG1+YlWcW1kr3ky++KlnMB9JnZSwBA8n4A8wEcuCjM7Bex+hsBHO/t7CJ1PPXsM4nrbiX73i6dt11Fam/Qba0+VJX6gphct3xwr5nFf75pAF6Jlfsxcu/mSwD+Ke3J2xT033+dIrUVKEB7W0j2I8V2JnHdlWRfd8EGr0jtLURb0/+mO9BkjLbRUgcNrzaSn0T1oviPqc/ehkL8/UeK1FagGO1tIdmPFNuZxLWGcSR3LfTsm+kHMD1WPh7A7sT5yD8AcCeA883sDV8nF4nz+FBVJnHd0k5VIh3zu1PVJgAzSZ5M8lAACwCsjVcgeQKA1QC+YGbP+/khRBrztMRxJnGde7InOY/kcyT7SF6b9/lHEm1Bt5fk9tixSSQ3kHwh+rPhFnV5Izmd5CMkd5LcQfJr0fEg2/tb1bVx0ryafpLZEIAlANYD2AlglZntILmY5OKo2n8HcCyAH5DcmtUWgiHHNaDYzkulUkn1GklWcc08HwKIphQ9j9iUIgAL66YUdQ3Js1Cds7rSzE6Jjv0VgH1mtiK6iI8xs2UjfU4eog02pprZUyQnANgC4LMALkOA7a2ZOGGazT31qlR1H37sv20JfV41EH5cA4rtPIwfP95mzJiRqu62bdtyj+28e/YHphSZ2fsAalOKgmBmjwLYV3d4PoB7o6/vRTXous7M9pjZU9HX+1HtAUxDoO09wMLbg9aDoOMaUGznIe0QTreess072TeaUjQt5za0aoqZ7QGqQQjgA11uTwLJkwCcCuBJFKC9oWxL6FER4xooQKwULbZDTvZ5z8ZJPaVI0iF5JKobZn/dzN6ubqsauPL9iyuuM1DE2A55bZy8e/apphQF5rXaBtTRn3u73J4DSI5F9WK4z8xWR4eDbW8NK5VUrwIpYlwDAcdKUWPbxw3arOSd7JtOKQrQWgCXRl9fCmBNF9tyAKvdnLsA7DSzm2LfCrK9BxiqD1WleRVHEeMaCDRWihrboY/Z573q5RDJ2pSiHgB3m9mOPNswEpJ/D+AcVB9l7gdwA4AVAFaR/BKAXQAu6l4LHWcC+AKAbSS3RseuQ7jtBQAQ5vOhqiCEHteAYjsvIQ/j5P4ErZmtA7Au7/OmYWYLD/Kt83JtSApm9jgajxUDAbbXEfAF0a6Q4xpQbOdFyV4kLuALQqQTSvYiNbUxe5ESUrIXiSnYTBuRVMzT5iVZUbKXnBXugSmR1NSzF6kxKNlLaSnZi8SF+5uuSEeU7EViyjbPXgTwunlJJpTsJX8BXxAinVCyF6kxA4Y1jiPlpNk4InEB935EOqGevUhcwBeESLs0Zi8SZwBS7C8rUkRK9iIHGGDhjmuKdELJXqTGoBu0Ulq6QSsSF3DvR6RdGrMXqRfwBSHSCSV7kQO0EJqUl5K9SI0BCHhcU6QTSvYicQFfECKdULIXOUDLJUg5afMSkTgDTPPspaTUsxeJ0xO0UlJK9iJxAV8QIp1QshepMdNsHCklPVQlUi/gC0KkE0r2IgcYbHi4240QyYRm44jUaIljKTH17EXiNPVSSij0Mfsx3W6AjC4GwCqW6pUGyXkknyPZR/LaBt8nye9H33+G5Ed9/0wiNbWE3+zVTBZxrWQv+bJo85I0ryZI9gC4DcD5AGYBWEhyVl218wHMjF6LANzu9wcS+S0fyT6ruFayl9zZ8HCqVwpzAfSZ2Utm9j6A+wHMr6szH8BKq9oI4GiSU/3+RCJVlUol1auJTOJaY/aSq/14c/3P7IHJKauPI7k5Vu41s95YeRqAV2LlfgCn1X1GozrTAOxJ2QaRtNYD8BHbmcS1kr3kyszmefw4NjpFG3VEOuYxtjOJaw3jSJH1A5geKx8PYHcbdURCkklcK9lLkW0CMJPkySQPBbAAwNq6OmsB/Hk0e+F0AP/PzDSEIyHLJK41jCOFZWZDJJegOlbaA+BuM9tBcnH0/R8CWAfgAgB9AN4FcHm32iuSRlZxzZAfAhARET80jCMiMgoo2YuIjAJK9iIio4CSvYjIKKBkLyIyCijZi4iMAkr2IiKjwP8HHGAJ9o1mLLgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "digit = train_X[9].reshape(28,28)\n",
    "plt.subplot(1,2,1)\n",
    "plt.imshow(digit)\n",
    "plt.colorbar()\n",
    "plt.subplot(1,2,2)\n",
    "plt.imshow(digit,cmap='gray')\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "继续输出该样本中的少数像素值（数据特征）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50000, 784)\n",
      "[0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.75       0.984375\n",
      " 0.73046875 0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.2421875  0.72265625 0.0703125  0.         0.         0.\n",
      " 0.         0.34765625 0.921875   0.84765625 0.18359375 0.\n",
      " 0.         0.        ]\n"
     ]
    }
   ],
   "source": [
    "print(train_X.shape)\n",
    "print(train_X[9][200:250])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(np.unique(train_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.7.2 用部分训练样本训练逻辑回归\n",
    "\n",
    "训练集的样本数目达到50000个，用整个训练集的样本进行训练，每次计算都需要消耗很多资源和时间，为了提高训练效率，可以从中取出部分数据，如用batch=500个样本进行训练："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w:  [[-0.1081236   0.15722483 -0.03458512 ...  0.14556747 -0.28445635\n",
      "  -0.0646967 ]\n",
      " [ 0.          0.          0.         ...  0.          0.\n",
      "   0.        ]\n",
      " [ 0.          0.          0.         ...  0.          0.\n",
      "   0.        ]\n",
      " ...\n",
      " [ 0.          0.          0.         ...  0.          0.\n",
      "   0.        ]\n",
      " [ 0.          0.          0.         ...  0.          0.\n",
      "   0.        ]\n",
      " [ 0.          0.          0.         ...  0.          0.\n",
      "   0.        ]]\n",
      "[2.291196965908743, 0.8358952420130783, 0.6737094231518491, 0.5552073298551986, 0.5205304526858188, 0.47774180354644386, 0.4642950154766653, 0.45662996516999915, 0.45159800707999115, 0.3368287901629059]\n"
     ]
    }
   ],
   "source": [
    "batch = 500\n",
    "\n",
    "alpha  =1e-2\n",
    "iterations  =1000\n",
    "reg = 1e-3\n",
    "\n",
    "w_history=[]\n",
    "\n",
    "w = np.zeros([train_X.shape[1]+1,len(np.unique(train_y))])\n",
    "for i in range(5):\n",
    "    s = i*batch\n",
    "    X = train_X[s :s+batch,:]\n",
    "    y = train_y[s :s+batch]\n",
    "    w_history_batch = gradient_descent_softmax(w,X,y,reg,alpha,iterations)\n",
    "    w = w_history_batch[-1]\n",
    "    w_history.extend(w_history_batch)\n",
    "    \n",
    "print(\"w: \",w)\n",
    "loss_history = compute_loss_history(w_history,X,y,reg)\n",
    "print(loss_history[:-1:len(loss_history)//10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "分别计算模型函数在训练集、验证集、测试集上的准确性："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练集的准确性： 0.86746\n",
      "验证集的准确性： 0.8837\n",
      "测试集的准确性： 0.8837\n"
     ]
    }
   ],
   "source": [
    "print(\"训练集的准确性：\",getAccuracy(w,train_X,train_y))\n",
    "print(\"验证集的准确性：\",getAccuracy(w,valid_X,valid_y))\n",
    "print(\"测试集的准确性：\",getAccuracy(w,test_X,test_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "绘制训练集和验证集的迭代损失学习曲线："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXwV9dn38c8FBEhYZAuCCQJSZFMERJC6PFotIi7Ulipt1erdPlTb3nXrYu+2alufahettVq9sdrWVqUVN6qoVatFK6iA7GAFBBIhAmGNLGG5nj9+c8ghnIQAmZwk832/XvOaOTNz5ly/oOc689vG3B0REUmuJtkOQEREskuJQEQk4ZQIREQSTolARCThlAhERBJOiUBEJOGUCOSQmNkCMzsji59/tJmVmVnTWr7uFWb2Rm1e8yA++3kz+3I2PluSrVm2A5CGyd0HpLbN7BbgE+5+aVyfZ2bLga+6+8vR568EWsf1edng7udmOwZJJt0RSNaZWaP/QdIYytgYyiCZKRHIITGz5WZ2tpmNAv4HuCSqqpkTHT/CzB40s9Vm9qGZ3ZqqxomqX/5tZr82s/XALWbWy8z+aWalZrbOzB4xs3bR+X8Gjgb+Hn3Gd82sh5m5mTUzs3FmNqNSfNeZ2eRou4WZ/crMVprZR2Z2v5nl1rCcfc3sJTNbb2bvmdnFacfOM7N3zWyzmRVFd0apY6n4vmJmK4F/pqqdolg2mNkHZnZu2nteM7Ovpv2Nqju3p5lNNbMtZvaymd1rZn+pphxjzGx2FOvS6N9t779j2nm3pK5TRRleMLNvVrr2HDP77IH+XlJ/KRHIYXH3F4CfAX9199bufkJ06E/ALuATwGBgJPDVtLcOB5YBnYH/BxhwG3AU0A/oBtwSfcZlwErggugzflEpjMlAHzPrnbbvi8Cj0fbPgWOBQVE8BcBNByqbmbUCXoqu0xn4AvA7M0tVi30MXA60A84Drjazz1S6zP+JynNOWrnfAzoBvwAeNDOrIoTqzn0UeBvoSPg7XVZNOYYBDwPfiWI9HVhebeGrLsOjhL9D6tr9ge7AczX4e0k9pUQgtc7MjgTOBa5194/dfQ3wa2Bc2mmr3P237r7L3be5+xJ3f8ndd7j7WuBOwhfQAbn7VuAZoi+oKCH0BSZHX5z/F7jO3de7+xZC4hpX1fXSnA8sd/c/RHHOAp4Axkaf+5q7z3P3Pe4+F3gsQ8y3RH+DbdHrFe7+gLvvJiTLrsCRVXx+xnPN7GjgJOAmdy939zcIybAqXwEeiv6+e9z9Q3dfXIPyZyrDU8AgM+seHfsS8KS77+AAfy+pv1TnJ3HoDuQAq9N+7DYBitLOSd/GzDoDdwOnAW2i8zccxGc+CtwB/IRwN/C0u2+NrpsHzEyLxYCa9DbqDgw3s41p+5oBf45iHg7cDhwHNAdaAI9XukZRpdclqY0oPqi60buqczsB66MEmP453aq4TjdgShXHamJvGdx9i5k9R0ikP4/W46PD1f69pP5SIpDaUHkK2yJgB9DJ3XfV8D23RfsGuntpVMVyTzXnV/YPoJOZDSLcGVwX7V8HbAMGuPuHB7hGZUXAv9z901UcfzSK8Vx3325mdxG+pNPFMb3vaqCDmeWlJYOqkgCEcvSq4tjHhESZ0iXDOZXL8Bhws5lNBXKBV9M+p7q/l9RTqhqS2vAR0MPMmgC4+2rCF/MdZtbWzJpEjcHVVfW0AcqAjWZWQKjPrvwZx1T15ijhTAJ+CXQg1FXj7nuAB4BfR3cHmFmBmZ1T1bXSPAsca2aXmVlOtJxkZv3SYl4fJYFhhDuR2Ln7CmAGoZG9uZmNAC6o5i0PAlea2VnRv0WBmfWNjs0GxkVlG0rNqnGmEH79/4TQNrQn2n+gv5fUU0oEUhtS1SGlZjYr2r6cUF2ykFDFM4lQx12VHwNDgE3Ac8CTlY7fBvzQzDaa2beruMajwNnA45XuRL4HLAGmm9lm4GWgz4EKFbUnjCRUf6wiVNX8nFAFBPB14CdmtoXQ+Py3A12zFn0JGAGUArcCfyXche3H3d8GriS002wC/kX4Igf4EeFuYQPh3+DRTNeodL0dhH+fs9PPr8HfS+op04NpRBo+M/srsNjdb852LNLw6I5ApAGKqlx6RVU9o4AxwNPZjksaptgSgZm1NLO3o8EmC8zsxxnOMTO728yWmNlcMxsSVzwijUwX4DVCu8rdwNXu/m5WI5IGK7aqoaj/dit3LzOzHOAN4Bp3n552zmjgv4HRhMEzv3H34bEEJCIiGcV2R+BBWfQyJ1oqZ50xwMPRudOBdmZWXYOiiIjUsljHEViYW2YmYVj/ve7+VqVTCth3wE1xtG91peuMJxq00qpVqxP79u2LiIjU3MyZM9e5e36mY7Emgmho/CALk4c9ZWbHufv8tFMyzbGyX12Vu08AJgAMHTrUZ8yYsd+bRESkama2oqpjddJryN03Ehq2RlU6VMy+IyILCf2PRUSkjsTZayjfKqYRziUMPqk80dVk4PKo99DJwKZoVKqIiNSROKuGugJ/itoJmgB/c/dnzewqAHe/nzBUfTRh1OdWwuhHERGpQ7Elgmha3sEZ9t+ftu3AN+KKQUQEYOfOnRQXF7N9+/ZshxK7li1bUlhYSE5OTo3fo9lHRaTRKy4upk2bNvTo0YOqnwPU8Lk7paWlFBcX07Nnzxq/T1NMiEijt337djp27NiokwCAmdGxY8eDvvNRIhCRRGjsSSDlUMqpRCAiknBKBCIiMdu4cSO/+93vDvp9o0ePZuPGjQc+8TApEYiIxKyqRLB79+5q3zdlyhTatWsXV1h7qdeQiEjMbrzxRpYuXcqgQYPIycmhdevWdO3aldmzZ7Nw4UI+85nPUFRUxPbt27nmmmsYP348AD169GDGjBmUlZVx7rnncuqpp/Lmm29SUFDAM888Q25ubq3Ep0QgIsly7bUwe3btXnPQILjrrioP33777cyfP5/Zs2fz2muvcd555zF//vy9XTwfeughOnTowLZt2zjppJP43Oc+R8eOHfe5xvvvv89jjz3GAw88wMUXX8wTTzzBpZdeWivhKxGIiNSxYcOG7dPP/+677+app54CoKioiPfff3+/RNCzZ08GDRoEwIknnsjy5ctrLR4lAhFJlmp+udeVVq1a7d1+7bXXePnll5k2bRp5eXmcccYZGccBtGjRYu9206ZN2bZtW63Fo8ZiEZGYtWnThi1btmQ8tmnTJtq3b09eXh6LFy9m+vTpGc+Lk+4IRERi1rFjR0455RSOO+44cnNzOfLII/ceGzVqFPfffz8DBw6kT58+nHzyyXUeX2zPLI6LHkwjIgdr0aJF9OvXL9th1JlM5TWzme4+NNP5qhoSEUk4JQIRkYRTIhARSTglAhGRhFMiEBFJOCUCEZGEUyIQEalnWrduDcCqVasYO3ZsxnPOOOMMaqsrvRKBiEg9ddRRRzFp0qTYP0cji0VEYva9732P7t278/Wvfx2AW265BTNj6tSpbNiwgZ07d3LrrbcyZsyYfd63fPlyzj//fObPn8+2bdu48sorWbhwIf369avVuYaUCEQkUbIwCzXjxo3j2muv3ZsI/va3v/HCCy9w3XXX0bZtW9atW8fJJ5/MhRdeWOUzh++77z7y8vKYO3cuc+fOZciQIbUWvxKBiEjMBg8ezJo1a1i1ahVr166lffv2dO3aleuuu46pU6fSpEkTPvzwQz766CO6dOmS8RpTp07lW9/6FgADBw5k4MCBtRafEoGIJEq2ZqEeO3YskyZNoqSkhHHjxvHII4+wdu1aZs6cSU5ODj169Mg4/XS6qu4WDldsjcVm1s3MXjWzRWa2wMyuyXDOGWa2ycxmR8tNccUjIpJN48aNY+LEiUyaNImxY8eyadMmOnfuTE5ODq+++iorVqyo9v2nn346jzzyCADz589n7ty5tRZbnHcEu4Ab3H2WmbUBZprZS+6+sNJ5r7v7+THGISKSdQMGDGDLli0UFBTQtWtXvvSlL3HBBRcwdOhQBg0aRN++fat9/9VXX82VV17JwIEDGTRoEMOGDau12GJLBO6+GlgdbW8xs0VAAVA5EYiIJMK8efP2bnfq1Ilp06ZlPK+srAwID6+fP38+ALm5uUycODGWuOpkHIGZ9QAGA29lODzCzOaY2fNmNqCK9483sxlmNmPt2rUxRioikjyxJwIzaw08AVzr7psrHZ4FdHf3E4DfAk9nuoa7T3D3oe4+ND8/P96ARUQSJtZEYGY5hCTwiLs/Wfm4u29297JoewqQY2ad4oxJRJKpoT2N8VAdSjnj7DVkwIPAIne/s4pzukTnYWbDonhK44pJRJKpZcuWlJaWNvpk4O6UlpbSsmXLg3pfnL2GTgEuA+aZWWoc3/8ARwO4+/3AWOBqM9sFbAPGeWP/lxKROldYWEhxcTFJaGNs2bIlhYWFB/WeOHsNvQFUO/rB3e8B7okrBhERgJycHHr27JntMOotzT4qIpJwSgQiIgmnRCAiknBKBCIiCadEICKScEoEIiIJp0QgIpJwiUkEHy3ewLM/X8DW0tp7zqeISGOQmEQw9X8XcsGNA1j2+ofZDkVEpF5JTCLo0iMXgJIlZVmORESkfklOIujdBoCS5dU/E1REJGmSkwj6dwCgpHhnliMREalfEpMIWndrTx4fU1JS7Tx4IiKJk5hEYE2b0KXpWkpK45x5W0Sk4UlMIgDo0mIDJRtzsx2GiEi9kqxE0KqM1R+3yXYYIiL1SrISQbttlOxon+0wRETqlWQlgo67WL+nPTt2ZDsSEZH6I1mJoGtYr1mhaSZERFKSlQgKQo+hksUbsxyJiEj9kaxE0LMlACVLtmQ5EhGR+iNZieATocfQ6g80zYSISEqyEkHfdhh7WLVyd7ZDERGpNxKVCHIKj6QLJRR9qGkmRERSEpUIyMujsOlqita0yHYkIiL1RmyJwMy6mdmrZrbIzBaY2TUZzjEzu9vMlpjZXDMbElc8Kd3y1lO0QaOLRURS4rwj2AXc4O79gJOBb5hZ/0rnnAv0jpbxwH0xxgNAt3ZbKNraAfe4P0lEpGGILRG4+2p3nxVtbwEWAQWVThsDPOzBdKCdmXWNKyaAbkeW8/GePDZtivNTREQajjppIzCzHsBg4K1KhwqAorTXxeyfLDCz8WY2w8xmrF279rBi6VYYbgWKVuw5rOuIiDQWsScCM2sNPAFc6+6bKx/O8Jb9Km3cfYK7D3X3ofn5+YcVT7djcgAomq/RxSIiEHMiMLMcQhJ4xN2fzHBKMdAt7XUhsCrOmLr1bQVA0UI9xF5EBOLtNWTAg8Aid7+zitMmA5dHvYdOBja5++q4YgLoelxHmrKLoiUaXSwiAhDncxtPAS4D5pnZ7Gjf/wBHA7j7/cAUYDSwBNgKXBljPAA07XYUR7GKopXqNiQiAjEmAnd/g8xtAOnnOPCNuGLIqEsXuvEWK1YfVacfKyJSXyVrZDFAs2Yc03I1H6xrm+1IRETqheQlAqBXu1KKPtaTykREIKmJoOtWnCYsX57tSEREsi+RieCYXqHpYukSNRiLiCQyEfQ6LheAZfO3ZjkSEZHsS2QiOPL4zuTxMUvnalCZiEgiE4H16M4xLGPpfzTfkIhIIhMBPXrQi6UsK87JdiQiIlmXzETQvj29copYtq6tnksgIomXzERgxifyN7FtV3M+/DDbwYiIZFcyEwHQr3voMbRoUZYDERHJsuQmgv5hLMHChVkOREQkyxKbCDr37UAHSln4ruaZEJFkS2wisE/0oh+LWDS3PNuhiIhkVWITAcceS38WsvB9dSEVkWRLbiLo1Yt+LKa0rCVr12Y7GBGR7EluImjRgv5HlgJqMBaRZEtuIgD69w1TTMyfn+VARESyKNGJoHBgBzqwntnvanixiCRXohOB9TmWIcxk1ts7sx2KiEjWJDoRcOyxDGEW8xY1o1y9SEUkoZKdCPr0YQiz2LmriRqMRSSxkp0ICgsZkrsYgFmzshyLiEiWJDsRNGlCr4GtaNP0YyUCEUmsZCcCoMkJxzPYZvPOO+o5JCLJFFsiMLOHzGyNmWXspW9mZ5jZJjObHS03xRVLtQYO5JO7pjJrFmzVs+xFJIHivCP4IzDqAOe87u6DouUnMcZStYEDOZU32LXLeOedrEQgIpJVsSUCd58KrI/r+rXm+OMZwTQA3ngjy7GIiGRBttsIRpjZHDN73swGVHWSmY03sxlmNmNtbc8Q164dHY5uw4Ajivj3v2v30iIiDUGNEoGZXWNmbS140MxmmdnIw/zsWUB3dz8B+C3wdFUnuvsEdx/q7kPz8/MP82MzGDiQU5u8yZtvwu7dtX95EZH6rKZ3BP/l7puBkUA+cCVw++F8sLtvdveyaHsKkGNmnQ7nmofspJM4dcOzbNoEc+dmJQIRkaypaSKwaD0a+IO7z0nbd0jMrIuZWbQ9LIql9HCueciGD+csXgbgxRezEoGISNbUNBHMNLN/EBLBi2bWBthT3RvM7DFgGtDHzIrN7CtmdpWZXRWdMhaYb2ZzgLuBce6enc78J51EV0oY2OUj/vGPrEQgIpI1zWp43leAQcAyd99qZh0I1UNVcvcvHOD4PcA9Nfz8eHXoAL17c46/yV1vXERZGbRune2gRETqRk3vCEYA77n7RjO7FPghsCm+sLJg+HDOKX2UnTvhtdeyHYyISN2paSK4D9hqZicA3wVWAA/HFlU2DB/OqRsm0ypvD889l+1gRETqTk0Twa6o/n4M8Bt3/w3QJr6wsuC002hBOaMHrOTpp9WNVESSo6aJYIuZfR+4DHjOzJoCOfGFlQXHHw8dOzI27zlKSuDNN7MdkIhI3ahpIrgE2EEYT1ACFAC/jC2qbGjSBM44g9FL76FlS2fSpGwHJCJSN2qUCKIv/0eAI8zsfGC7uzeuNgKAM8+kdfFizj19K5MmqXpIRJKhplNMXAy8DXweuBh4y8zGxhlYVpx5JgDjer7FqlXwyitZjkdEpA7UtGroB8BJ7v5ld78cGAb8KL6wsqRfP+jalTFrHqBjR3jwwWwHJCISv5omgibuvibtdelBvLfhMIPzz6fFy89x6Rd28fTTUJqdSS9EROpMTb/MXzCzF83sCjO7AngOmBJfWFl04YWwZQtfOe5tysvh4cbXEiIiso+aNhZ/B5gADAROACa4+/fiDCxrzjoLcnM5ft6jnHIK3H037NqV7aBEROJT4+odd3/C3a939+vc/ak4g8qq3FwYORImT+bbNzjLl8MTT2Q7KBGR+FSbCMxsi5ltzrBsMbPNdRVknbvoIigq4sL8afTuDb/8JWRpXlQRkdhVmwjcvY27t82wtHH3tnUVZJ276CLIzaXJY4/w7W/DzJnw/PPZDkpEJB6Nr+dPbWjbFsaMgb/+lSu+WM4xx8D3vw97qn0Cg4hIw6REUJVLL4XSUpr/8wVuvTU8wvKxx7IdlIhI7VMiqMrIkdClC0yYwCWXwKBB4a6grCzbgYmI1C4lgqrk5MDXvgZTptDkg6Xcey8UFcHNN2c7MBGR2qVEUJ3x46FpU7jvPj75yZAX7roL3n0324GJiNQeJYLqHHUUfO5zYdKhzZu57Tbo3Bkuuwy2bs12cCIitUOJ4EC+/W3YuBHuvZf27cOUEwsWwA03ZDswEZHaoURwIEOHwujRcMcdUFbGpz8N3/0u3H+/ehGJSOOgRFATN90UpiG95x4AfvpTOO00uPJKmD49y7GJiBwmJYKaGD4czjsPbrsN1qyheXN48kkoKAjjzpYty3aAIiKHTomgpu64I7QQ/+AHAHTqBM8+Czt3wqc+BStXZjk+EZFDFFsiMLOHzGyNmc2v4riZ2d1mtsTM5prZkLhiqRV9+sC3vhV6EM2YAYQHmr30UmhLPvPMMM5ARKShifOO4I/AqGqOnwv0jpbxwH0xxlI7broJunaFK66AHTsAOPFEePFFWLcORoyAefOyG6KIyMGKLRG4+1RgfTWnjAEe9mA60M7MusYVT6044ohwR7BgAdxyy97dw4fD66+H7VNPDXcJIiINRTbbCAqA9MqU4mjffsxsvJnNMLMZa9eurZPgqjRqFHz1q/CLX8Brr+3dPXAgTJsGRx8dTrn1Vs1WKiINQzYTgWXYl/HxL+4+wd2HuvvQ/Pz8mMOqgTvvDG0GF18MxcV7d3frFpLBuHHwox+F4QerV2cxThGRGshmIigGuqW9LgRWZSmWg9OmTeg/un17mIJi27a9h1q3hr/8Be67D/71L+jfH/70Jz3hTETqr2wmgsnA5VHvoZOBTe7ecH4/9+0b5pt45x245JJ9nnBvBlddBXPmwIABoW151ChYuDB74YqIVCXO7qOPAdOAPmZWbGZfMbOrzOyq6JQpwDJgCfAA8PW4YonNZz4TRhv//e9hptJKjQLHHgtTp8Ldd8Nbb4V2hG98I/QwEhGpL8wbWJ3F0KFDfUbUj7/euOUW+PGPw5wTDzwQpq6uZO3acNr//i/k5oaEcP31YTZTEZG4mdlMdx+a6ZhGFteGm28O3/J/+AN84QtQXr7fKfn5cO+94ZGX558fOh316AHXXQcffFDnEYuI7KVEUBvMQjK44w54/HE46yxYsybjqf37h1lLFy2Cz38efvtb6NULLrgAXnhBXU5FpO4pEdSm66+HiRNh5swwffWsWVWe2qdP6E20fDn88Iehzfncc0NS+NGP4L336i5sEUk2JYLadskl8MYbYfvkk+FXv6r2Z35hIfzkJ2HSukcfDQ3MP/tZ6JQ0bBj8+tea3VRE4qVEEIchQ8KDjS+4AL7zHfj0pw/YENC8eWheePHFMEbtzjtDj9Trrw93CccfH+4c3n5b1UciUrvUayhO7qEB+Zprwrf6D38YHn3ZokWNL7FsGUyeDM88E+Yz2r0bOnQIs52edVZYevcOzRQiIlWprteQEkFdKC4OP+0ffzx8a992G3z2swf97b1+PTz/PLz8MrzySsW014WFcPrpYfbTESPCeIWcnBjKISINlhJBffHii3DttbB4cZi/+mc/C9VGh/Bz3h2WLAkJ4ZVX4N//rpjXKDc3tFWPGBG6qp56qu4YRJJOiaA+2b0b/vzn0N105crQInzDDeEOoVmzQ76se7jc9Olh4rtp00Izxc6dcM45oUF62LBaLIeINCgaUFafNG0aJh/6z3/CzHQbNoSeRr17hxbiQ5x/wgy6dw+XuuuuMKXF+vWh19H06eGZCSNGhI/M9kzeIlK/KBFkS4sWYWa6RYvgqadCRf8NN8BRR4WRZi+8EO4eDkPr1qEmqqgozHe0aRN8/evhIWsjR4aEsXixZkYVSTpVDdUnCxaEJ6A9/DCUlkKXLmGa689/PlT0Z5jD6GC4h0dpTpwITzwRbkogPEzn7LPhk58MS58+0EQ/EUQaFbURNDQ7doQ+oxMnwpQp4bkHXbqE2U5Hj4ZPfQpatTrsj1m+PLRfv/BCmCV1ffRg0fbtw1i4QYNCD6QTTgg1V4fRhCEiWaZE0JCVlcFzz4Wup88/D1u3htFnp50W5qQYOTI89OAwf8Lv2RPuEKZNgzffDG0MixZVPGahRYvwMX36wCc+ERJD795hu2NH9UoSqe+UCBqLHTvC9BXPPx9+xi9YEPa3bx+qjk47LSwnnlgrAwnKy0MymDu3Ynn/fVixYt/Rze3ahZlUu3XLvBQUhNwlItmjRNBYFRXBq6+Gep3XX6+o9M/Lg1NOCcOOR46EwYNr9WN37AgzZixZEhJDKjkUFYWxcxs27P+e9u3hyCPD8xeqWnfoEM5r316JQ6S2KREkRUlJuGP4179CgkjdMXz2s6Ef6dFH10kYZWUhIRQVVSSHNWvgo4/2XWdKGCl5eRVJIT1BpO9LLR07Vmy3bauGbpFMlAiSqqQEHnoIfvrT0BX1iivCVBd9+2Y7MiBUPa1dW5EY1q8PyeFAS1lZ1dds0iQ8BGjo0DCuomvX0M6evnTurDsOSR4lgqRbuRJ+/nP4/e/Dt+/pp8PXvhZ6IeXlZTu6g7ZzZ0gI69eHXrbr11cspaXw4YcwYwasWlXRE6qyjh1DlVSXLhXrTNv5+Yfda1ekXlAikOCjj+CPf4QJE8K0pnl5oTvq2LFh3aZNtiOsdTt2hLuNkpJ9l9Wrw5+jpCSsV68OHbIqa9Jk/yqqqtbt2oU/YWpp3Tr08lWPKqkPlAhkX3v2hHaExx+HJ58M34TNmoU5KM4+O0yEN3Ro4qYwLSurSA6pBFFSUnHXkboLSa03bjzwsyHMQkJITxDpS15emCQwN3ff7cpLpmN5eeEaumORmlAikKrt3h2mLn3+eXjppfB4TffwTTNkSJikaPjwMGNd9+76eZtmzx7YsqUiOWzcGF6nlrKyfV9nWrZuhW3bwlJefmhxtG0b7kp69gxPuDv22PAwo6OPDt13Nc5DQIlADkZpKfzznxWjymbNCvUrEL5xBgyA444Ly4AB4VunoEBddWrB7t1hEPm2bfsmiMpL6tjWrbB5c0XbyLJloQdxaem+183NDQnh6KPDUlgY2j9SS6oLb9u2ShiNmRKBHLry8jCSbMYMmD8/LPPm7dsK26JF+Dnaqxccc0xY9+gREkRBQfimUf1FnSktDdOHrFwZlqKifbdXr8480WCLFvuO6WjXrqLto/K6XTs44oiKdpBWrRJXk9jgKBFI7XIPFejz54dRZUuXhp+jS5eGpXL/zmbNQjecVGIoKAivO3UK3XLy8yu227fX3UXMdu0K3XZTYzrSx3ekj/FIVXdt2BB6ah1I8+YhIaQnh9R269bhzqRFC2jZ8uDXubnhOnl5FdfWb4uDk7VEYGajgN8ATYHfu/vtlY6fATwDpJ7s/qS7/6S6ayoR1HPu4ZkKK1aEfpxVLZs3Z35/kyahUjs/P6zTf35Wt27XLvyM1cx4tc49VEWlJ4aNG8O05h9/HJaysn3XmfZt2xZqGbdvD+sdOw5vCvQ2bcIdjHtor0ktqdcQklP60qLF/vtSS05O+M8n07q6Y5XPadIk82JW9bGaHm/fPvxmOhTVJYLY/q8xs6bAvcCngWLgHTOb7O4LK536urufH1ccUsfMKn7lD83431ywfXtIGGvX7r9Oba9bF+oy5s0L3zqbNlXfTccsJI9eveCLXwzzL/XpUysztYO/VPkAAAwJSURBVCaZWfglnpcXbuZqi3u406icHFLb6ett2yoSTGopLQ13L6kvyqZN9/0yTV2/vDxcp7y8Yikrq9hOHdu1K5yfWqdv1xff+x7cfvuBzztYcf58GgYscfdlAGY2ERgDVE4EkkQtW4ZWy8LCmr/HPfwfnPo5unFjxfaGDRXDlKdPh2uuqXhffn5ow0hVSWVa8vND/YPUGbOKX+P1eQhL6g4jPTFkShapdeU7lMrLoRxPPaOqf/94yhhnIigAitJeFwPDM5w3wszmAKuAb7v7gsonmNl4YDzA0XU0X47UQ2YVHfC7dav6PPfQWvrOO6HN4oMPwuv33w+T81X1ONAWLSomLao8YizVepoaFJBp3bp1SHDqetOomIW7jcbcJhFnIsj0f0PlGsFZQHd3LzOz0cDTQO/93uQ+AZgAoY2gtgOVRsYs3AH07Jn5+M6dFcONV68O63Xr9h81tmIFvPvugSc4Ste0aUVyyMuraOnMza3Yrmqdvp2qtK5cCZ3pdU5OKPOWLeEOaevWijoSs3C8qpFqublqV5FYE0ExkP6zrZDwq38vd9+ctj3FzH5nZp3c/dCe4C5SEzk5Fb2Xaqq8PFRBlZVVjBQ70DrV4T9VyV1auu/r1Hrbtuw+OLpZs/2T0YFaWCvvy8mp+NnctGlFpX1V21Udz/S6qmXQoJB05bDFmQjeAXqbWU/gQ2Ac8MX0E8ysC/CRu7uZDQOaAKX7XUkk25o3r2gEr22pVs305FBevn9ldKbXqX179oS7kHbtwp2I+76V2+kj0tITUKblQK2rmY6Xl4eK7N27DzzvRm059tjQHtS+fd18XiMWWyJw911m9k3gRUL30YfcfYGZXRUdvx8YC1xtZruAbcA4b2gDG0QOV3qr6RFHZDua2pFq4Uyta7qd6XWmpbgYvvrV8ACmhx+uvoeaHJAGlIlIw/TqqzBuXGjvOe+8kBjOOUe9v6pQ3TgCDeEUkYbpzDPD5Eo33xzmxLroojCO5MIL4d57w9QoqX6XUi3dEYhIw7drF7zyCjz3HPz976G7MISqtk9+MkyxPnhwmFG3a9dEdvHVXEMikhzuoevvG29ULAvShid17hySwuDBYRbdfv0SMQJdiUBEkm3zZpgzJ4wLSS0LFoQ7iZTu3UNSSC2pBzscdVSjmAgxK3MNiYjUG23bwmmnhSWlvDyMNl+0qGJZvDg8vW/btorzmjcPgxOPOaZiSU253r17uHYDp0QgIsnUvHl4uNKAAfvu37MnPLzh/ffD9OqpKdaXLQtP86s8c25qypPCwv3Xqe16niyUCERE0jVpEh6s1KPH/sfcw5QjqeSwcmUY01BcXDFTbknJ/iPFW7cOjdSpSQ6PPHLfSQ9Trzt3DgmqjikRiIjUlFnFJIRVDWIrLw9zWBUVVSSI4uIwM25JSejW+tFHYV6oTDp2rEgKqdHsqWXYsFgGzykRiIjUpubNQ9tB9+7Vn7d9e0VyKCnZd3v16jCt+rvvhnUqaXz/+0oEIiKNRsuWNUsYEOaLWrcutgdDKxGIiNR3OTmhjSEmDb9zrIiIHBYlAhGRhFMiEBFJOCUCEZGEUyIQEUk4JQIRkYRTIhARSTglAhGRhFMiEBFJOCUCEZGEUyIQEUk4JQIRkYRTIhARSTglAhGRhIs1EZjZKDN7z8yWmNmNGY6bmd0dHZ9rZkPijEdERPYXWyIws6bAvcC5QH/gC2bWv9Jp5wK9o2U8cF9c8YiISGZx3hEMA5a4+zJ3LwcmAmMqnTMGeNiD6UA7M4vv6QsiIrKfOBNBAVCU9ro42new52Bm481shpnNWLt2ba0HKiKSZHEmAsuwzw/hHNx9grsPdfeh+fn5tRKciIgEcSaCYqBb2utCYNUhnCMiIjGKMxG8A/Q2s55m1hwYB0yudM5k4PKo99DJwCZ3Xx1jTCIiUkmzuC7s7rvM7JvAi0BT4CF3X2BmV0XH7wemAKOBJcBW4Mq44hERkcxiSwQA7j6F8GWfvu/+tG0HvhFnDCIiUj2NLBYRSTglAhGRhFMiEBFJOCUCEZGEUyIQEUk4JQIRkYRTIhARSTglAhGRhLMwpqvhMLO1wIpDfHsnYF0thtMQqMzJoDInw+GUubu7Z5y1s8ElgsNhZjPcfWi246hLKnMyqMzJEFeZVTUkIpJwSgQiIgmXtEQwIdsBZIHKnAwqczLEUuZEtRGIiMj+knZHICIilSgRiIgkXGISgZmNMrP3zGyJmd2Y7XgOh5k9ZGZrzGx+2r4OZvaSmb0frdunHft+VO73zOyctP0nmtm86NjdZmZ1XZaaMLNuZvaqmS0yswVmdk20vzGXuaWZvW1mc6Iy/zja32jLnGJmTc3sXTN7NnrdqMtsZsujWGeb2YxoX92W2d0b/UJ4VOZS4BigOTAH6J/tuA6jPKcDQ4D5aft+AdwYbd8I/Dza7h+VtwXQM/o7NI2OvQ2MAAx4Hjg322WrorxdgSHRdhvgP1G5GnOZDWgdbecAbwEnN+Yyp5X9euBR4NnG/t92FOtyoFOlfXVa5qTcEQwDlrj7MncvByYCY7Ic0yFz96nA+kq7xwB/irb/BHwmbf9Ed9/h7h8Qng89zMy6Am3dfZqH/4oeTntPveLuq919VrS9BVgEFNC4y+zuXha9zIkWpxGXGcDMCoHzgN+n7W7UZa5CnZY5KYmgAChKe10c7WtMjnT31RC+OIHO0f6qyl4QbVfeX6+ZWQ9gMOEXcqMuc1RFMhtYA7zk7o2+zMBdwHeBPWn7GnuZHfiHmc00s/HRvjotc6wPr69HMtWVJaXfbFVlb3B/EzNrDTwBXOvum6upAm0UZXb33cAgM2sHPGVmx1VzeoMvs5mdD6xx95lmdkZN3pJhX4Mqc+QUd19lZp2Bl8xscTXnxlLmpNwRFAPd0l4XAquyFEtcPopuD4nWa6L9VZW9ONquvL9eMrMcQhJ4xN2fjHY36jKnuPtG4DVgFI27zKcAF5rZckL17afM7C807jLj7qui9RrgKUJVdp2WOSmJ4B2gt5n1NLPmwDhgcpZjqm2TgS9H218GnknbP87MWphZT6A38HZ0u7nFzE6OehdcnvaeeiWK70FgkbvfmXaoMZc5P7oTwMxygbOBxTTiMrv799290N17EP4f/ae7X0ojLrOZtTKzNqltYCQwn7ouc7ZbzOtqAUYTepssBX6Q7XgOsyyPAauBnYRfAl8BOgKvAO9H6w5p5/8gKvd7pPUkAIZG/9EtBe4hGmle3xbgVMJt7lxgdrSMbuRlHgi8G5V5PnBTtL/RlrlS+c+gotdQoy0zoSfjnGhZkPpuqusya4oJEZGES0rVkIiIVEGJQEQk4ZQIREQSTolARCThlAhERBJOiUASx8zejNY9zOyLtXzt/8n0WSL1mbqPSmJF0xh8293PP4j3NPUw9UNVx8vcvXVtxCdSV3RHIIljZqlZPW8HTovmgb8umuTtl2b2jpnNNbOvReefYeF5CI8C86J9T0eThC1ITRRmZrcDudH1Hkn/LAt+aWbzoznjL0m79mtmNsnMFpvZI6l55M3sdjNbGMXyq7r8G0myJGXSOZFMbiTtjiD6Qt/k7ieZWQvg32b2j+jcYcBxHqb+Bfgvd18fTf/wjpk94e43mtk33X1Qhs/6LDAIOAHoFL1nanRsMDCAMDfMv4FTzGwhcBHQ1909Nd2ESBx0RyBSYSRweTT181uEYf69o2NvpyUBgG+Z2RxgOmESsN5U71TgMXff7e4fAf8CTkq7drG77yFMn9ED2AxsB35vZp8Fth526USqoEQgUsGA/3b3QdHS091TdwQf7z0ptC2cDYxw9xMIcwK1rMG1q7IjbXs30MzddxHuQp4gPGDkhYMqichBUCKQJNtCePRlyovA1dGU15jZsdGMkJUdAWxw961m1pfwCMmUnan3VzIVuCRqh8gnPG707aoCi569cIS7TwGuJVQricRCbQSSZHOBXVEVzx+B3xCqZWZFDbZryfy4vxeAq8xsLmEGyOlpxyYAc81slrt/KW3/U4Tnyc4hzKT6XXcviRJJJm2AZ8ysJeFu4rpDK6LIgan7qIhIwqlqSEQk4ZQIREQSTolARCThlAhERBJOiUBEJOGUCEREEk6JQEQk4f4/pgRObkmi1QQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss_history_valid = compute_loss_history(w_history,valid_X[0:1000,:],valid_y[0:1000],reg)\n",
    "\n",
    "plt.plot(loss_history, color='r')\n",
    "plt.plot(loss_history_valid, color='b') \n",
    "plt.ylim(0,5)\n",
    "plt.xlabel('iterations')\n",
    "plt.ylabel('loss')\n",
    "plt.title('iterative learning curve')\n",
    "plt.legend(['train', 'valid'])\n",
    "plt.ylim(-0.2,3)\n",
    "plt.show()      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.7.3 批梯度下降法及实现\n",
    "\n",
    "批梯度下降法的一般做法是：\n",
    "+ 1) 对原训练集中的样本次序重新排列，即打乱原训练集中样本的次序。\n",
    "+ 2) 对重排后的训练集，从头开始，依次取出一小批样本，用这一批样本计算模型函数损失的梯度，并更新模型参数。\n",
    "+ 3) 多次重复 1),2)。\n",
    "上述中的1),2)完成了对训练集中几乎所有样本的一趟遍历，在这一趟遍历中用不同的一小批样本对模型参数进行了更新。因此，这个1),2)的过程称为一个epoch。3)表示执行多次的epoch。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "打乱一个列表的次序可用numpy.random.shuffle()函数，例如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 2, 3, 4]\n",
      "[2, 1, 0, 3, 4]\n"
     ]
    }
   ],
   "source": [
    "m=5\n",
    "indices = list(range(m))\n",
    "print(indices)\n",
    "np.random.shuffle(indices)\n",
    "print(indices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对应一个数据集(X,y)，可定义一个迭代器函数data_iter()，用于打乱原数据集的次序，并每次从数据集中返回batchsize大小的一小批训练样本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_iter(X,y,batch_size,shuffle=False):\n",
    "    m = len(X)  \n",
    "    indices = list(range(m))\n",
    "    if shuffle:                 # shuffle是True表示打乱次序\n",
    "        np.random.shuffle(indices)\n",
    "    for i in range(0, m - batch_size + 1, batch_size):\n",
    "        batch_indices = np.array(indices[i: min(i + batch_size, m)])      \n",
    "        yield X.take(batch_indices,axis=0), y.take(batch_indices,axis=0)   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "批梯度下降法的代码实现："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "def batch_gradient_descent_softmax(w,X, y, epochs,batchsize = 50,shuffle = False,\n",
    "                                   reg=0., alpha=0.01, gamma = 0.8,epsilon=1e-8): \n",
    "    w_history = []\n",
    "    X = np.hstack((np.ones((X.shape[0], 1), dtype=X.dtype),X)) \n",
    "    for epoch in range(epochs):\n",
    "        for X_batch,y_batch in data_iter(X,y,batchsize,shuffle):\n",
    "            gradient = gradient_softmax(w,X_batch,y_batch,reg)\n",
    "            if np.max(np.abs(gradient))<epsilon:\n",
    "                print(\"gradient is small enough!\")\n",
    "                print(\"iterated num is :\",i)\n",
    "                break          \n",
    "            w = w - (alpha * gradient)\n",
    "            w_history.append(w)\n",
    "    return w_history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对Mnist手写数字识别训练集，执行这个批梯度下降法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w:  [[-0.09799915  0.19013309 -0.03203123 ...  0.14410694 -0.35215659\n",
      "  -0.05804954]\n",
      " [ 0.          0.          0.         ...  0.          0.\n",
      "   0.        ]\n",
      " [ 0.          0.          0.         ...  0.          0.\n",
      "   0.        ]\n",
      " ...\n",
      " [ 0.          0.          0.         ...  0.          0.\n",
      "   0.        ]\n",
      " [ 0.          0.          0.         ...  0.          0.\n",
      "   0.        ]\n",
      " [ 0.          0.          0.         ...  0.          0.\n",
      "   0.        ]]\n",
      "[2.293262307414389, 0.825868886995949, 0.643869466832342, 0.570418897469662, 0.5338953881876869, 0.5070184011149137, 0.4900678967579153, 0.47706039807365735, 0.46572411077892795, 0.4613240435830291]\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "batchsize = 50\n",
    "epochs = 5\n",
    "shuffle = True\n",
    "alpha = 0.01\n",
    "reg = 1e-3\n",
    "gamma = 0.8\n",
    "\n",
    "X,y = train_X,train_y\n",
    "w = np.zeros([X.shape[1]+1,len(np.unique(y))])   \n",
    "w_history = batch_gradient_descent_softmax(w,train_X,train_y,epochs,batchsize,\n",
    "                                                shuffle,reg,alpha,gamma)\n",
    "w = w_history[-1]\n",
    "print(\"w: \",w)\n",
    "X,y = train_X[0:1000,:],train_y[0:1000]\n",
    "loss_history = compute_loss_history(w_history,X,y,reg)\n",
    "print(loss_history[:-1:len(loss_history)//10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面输出模型在不同样本集上的准确度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练集的准确性： 0.89342\n",
      "验证集的准确性： 0.9056\n",
      "测试集的准确性： 0.9056\n"
     ]
    }
   ],
   "source": [
    "print(\"训练集的准确性：\",getAccuracy(w,train_X,train_y))\n",
    "print(\"验证集的准确性：\",getAccuracy(w,valid_X,valid_y))\n",
    "print(\"测试集的准确性：\",getAccuracy(w,test_X,test_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "绘制迭代学习曲线："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXxddZ3/8dcnNzd7uqdbugqFslhKbQsMDFZxlBa0jjJSF2AYZzqgjsDPGWFmEHEZB3VmVNwYUFSGbbTIIlYQFCg4LLa1S2hZWmhpkpakaZsmzZ58fn+cE3IJSZq2OTlpzvv5eNzHPfds9/O9hfvO93uWa+6OiIgkV1bcBYiISLwUBCIiCacgEBFJOAWBiEjCKQhERBJOQSAiknAKAjksZva8mS2K8f2nmVm9maUGeL9/bWZPDeQ+D+G9f2Nml8Tx3pJs2XEXIEcndz+pc9rMrgeOdfdPRPV+ZrYN+Ft3fzR8/9eAoqjeLw7uvjjuGiSZ1COQ2JnZsP+DZDi0cTi0QXqmIJDDYmbbzOw9ZnYu8C/AheFQzfpw+Ugz+7GZ7TSzCjP7aucwTjj88gcz+5aZ7QGuN7NjzOz3ZlZjZrvN7A4zGxWu/z/ANOBX4Xt83sxmmJmbWbaZLTOz1d3qu8rMHginc83sP8zsNTN73cxuMrP8frZztpk9YmZ7zOxFM/tIxrLzzOxPZrbfzHaEPaPOZZ31fdLMXgN+3znsFNay18xeNbPFGds8bmZ/m/EZ9bXuTDNbZWZ1ZvaomX3fzG7vox1LzWxdWOvW8N/tjX/HjPWu79xPL214yMw+023f683sQwf7vGToUhDIEXH3h4CvAf/r7kXufkq46GdAG3AscCrwXuBvMzY9DXgFGA/8G2DAvwOTgROAqcD14XtcBLwGvD98j290K+MB4Hgzm5Ux72PAneH014HjgLlhPaXAdQdrm5kVAo+E+xkPfBT4gZl1DosdAC4GRgHnAZeb2Qe77eadYXvel9HuF4FxwDeAH5uZ9VJCX+veCTwHjCX4nC7qox0LgduAfwprPRvY1mfje2/DnQSfQ+e+TwSmA7/ux+clQ5SCQAacmU0AFgNXuvsBd68CvgUsy1it0t2/6+5t7t7o7lvc/RF3b3b3auC/CL6ADsrdG4D7Cb+gwkCYDTwQfnH+HXCVu+9x9zqC4FrW2/4ynA9sc/efhHWuBe4BLgjf93F33+juHe6+Abirh5qvDz+DxvD1dne/xd3bCcJyEjChl/fvcV0zmwYsAK5z9xZ3f4ogDHvzSeDW8PPtcPcKd3+hH+3vqQ33AnPNbHq47OPAL929mYN8XjJ0acxPojAdSAM7M/7YzQJ2ZKyTOY2ZjQduBP4cKA7X33sI73kn8J/Alwl6A/e5e0O43wJgTUYtBvTnbKPpwGlmti9jXjbwP2HNpwE3ACcDOUAu8Itu+9jR7fWuzomwPuj9oHdv644D9oQBmPk+U3vZz1RgZS/L+uONNrh7nZn9miBIvx4+Lw8X9/l5ydClIJCB0P0WtjuAZmCcu7f1c5t/D+fNcfeacIjle32s391vgXFmNpegZ3BVOH830Aic5O4VB9lHdzuAJ9z9L3pZfmdY42J3bzKzbxN8SWeK4va+O4ExZlaQEQa9hQAE7Timl2UHCIKy08Qe1unehruAL5rZKiAfeCzjffr6vGSI0tCQDITXgRlmlgXg7jsJvpj/08xGmFlWeDC4r6GeYqAe2GdmpQTj2d3f4229bRwGzgrgm8AYgrFq3L0DuAX4Vtg7wMxKzex9ve0rw4PAcWZ2kZmlw8cCMzsho+Y9YQgsJOiJRM7dtwOrCQ6y55jZGcD7+9jkx8ClZnZO+G9Ramazw2XrgGVh2+bTv2GclQR//X+Z4NhQRzj/YJ+XDFEKAhkIncMhNWa2Npy+mGC4ZBPBEM8KgjHu3nwJmAfUAr8Gftlt+b8D15rZPjP7x172cSfwHuAX3XoiVwNbgGfMbD/wKHD8wRoVHk94L8HwRyXBUM3XCYaAAD4FfNnM6ggOPv/8YPscQB8HzgBqgK8C/0vQC3sLd38OuJTgOE0t8ATBFznAFwh6C3sJ/g3u7Gkf3fbXTPDv857M9fvxeckQZfphGpGjn5n9L/CCu38x7lrk6KMegchRKBxyOSYc6jkXWArcF3ddcnSKLAjMLM/MngsvNnnezL7UwzpmZjea2RYz22Bm86KqR2SYmQg8TnBc5Ubgcnf/U6wVyVErsqGh8PztQnevN7M08BRwhbs/k7HOEuAfgCUEF898x91Pi6QgERHpUWQ9Ag/Uhy/T4aN76iwFbgvXfQYYZWZ9HVAUEZEBFul1BBbcW2YNwWX933f3Z7utUsqbL7gpD+ft7Laf5YQXrRQWFr5j9uzZiIhI/61Zs2a3u5f0tCzSIAgvjZ9rwc3D7jWzk929LGOVnu6x8paxKne/GbgZYP78+b569eq3bCQiIr0zs+29LRuUs4bcfR/Bga1zuy0q581XRE4hOP9YREQGSZRnDZVY122E8wkuPul+o6sHgIvDs4dOB2rDq1JFRGSQRDk0NAn4WXicIAv4ubs/aGaXAbj7TQSXqi8huOqzgeDqRxERGUSRBUF4W95Te5h/U8a0A5+OqgYREYDW1lbKy8tpamqKu5TI5eXlMWXKFNLpdL+30d1HRWTYKy8vp7i4mBkzZtD77wAd/dydmpoaysvLmTlzZr+30y0mRGTYa2pqYuzYscM6BADMjLFjxx5yz0dBICKJMNxDoNPhtFNBICKScAoCEZGI7du3jx/84AeHvN2SJUvYt2/fwVc8QgoCEZGI9RYE7e3tfW63cuVKRo0aFVVZb9BZQyIiEbvmmmvYunUrc+fOJZ1OU1RUxKRJk1i3bh2bNm3igx/8IDt27KCpqYkrrriC5cuXAzBjxgxWr15NfX09ixcv5qyzzuL//u//KC0t5f777yc/P39A6lMQiEiyXHklrFs3sPucOxe+/e1eF99www2UlZWxbt06Hn/8cc477zzKysreOMXz1ltvZcyYMTQ2NrJgwQI+/OEPM3bs2Dft4+WXX+auu+7illtu4SMf+Qj33HMPn/jEJwakfAWBiMggW7hw4ZvO87/xxhu59957AdixYwcvv/zyW4Jg5syZzJ07F4B3vOMdbNu2bcDqURCISLL08Zf7YCksLHxj+vHHH+fRRx/l6aefpqCggEWLFvV4HUBubu4b06lUisbGxgGrRweLRUQiVlxcTF1dXY/LamtrGT16NAUFBbzwwgs888wzPa4XJfUIREQiNnbsWM4880xOPvlk8vPzmTBhwhvLzj33XG666SbmzJnD8ccfz+mnnz7o9UX2m8VR0Q/TiMih2rx5MyeccELcZQyantprZmvcfX5P62toSEQk4RQEIiIJpyAQEUk4BYGISMIpCEREEk5BICKScAoCEZEhpqioCIDKykouuOCCHtdZtGgRA3UqvYJARGSImjx5MitWrIj8fXRlsYhIxK6++mqmT5/Opz71KQCuv/56zIxVq1axd+9eWltb+epXv8rSpUvftN22bds4//zzKSsro7GxkUsvvZRNmzZxwgknDOi9hhQEIpIoMdyFmmXLlnHllVe+EQQ///nPeeihh7jqqqsYMWIEu3fv5vTTT+cDH/hAr785/MMf/pCCggI2bNjAhg0bmDdv3oDVryAQEYnYqaeeSlVVFZWVlVRXVzN69GgmTZrEVVddxapVq8jKyqKiooLXX3+diRMn9riPVatW8dnPfhaAOXPmMGfOnAGrT0EgIokS112oL7jgAlasWMGuXbtYtmwZd9xxB9XV1axZs4Z0Os2MGTN6vP10pt56C0cqsoPFZjbVzB4zs81m9ryZXdHDOovMrNbM1oWP66KqR0QkTsuWLePuu+9mxYoVXHDBBdTW1jJ+/HjS6TSPPfYY27dv73P7s88+mzvuuAOAsrIyNmzYMGC1RdkjaAM+5+5rzawYWGNmj7j7pm7rPenu50dYh4hI7E466STq6uooLS1l0qRJfPzjH+f9738/8+fPZ+7cucyePbvP7S+//HIuvfRS5syZw9y5c1m4cOGA1RZZELj7TmBnOF1nZpuBUqB7EIiIJMLGjRvfmB43bhxPP/10j+vV19cDwY/Xl5WVAZCfn8/dd98dSV2Dch2Bmc0ATgWe7WHxGWa23sx+Y2Yn9bL9cjNbbWarq6urI6xURCR5Ig8CMysC7gGudPf93RavBaa7+ynAd4H7etqHu9/s7vPdfX5JSUm0BYuIJEykQWBmaYIQuMPdf9l9ubvvd/f6cHolkDazcVHWJCLJdLT9GuPhOpx2RnnWkAE/Bja7+3/1ss7EcD3MbGFYT01UNYlIMuXl5VFTUzPsw8DdqampIS8v75C2i/KsoTOBi4CNZtZ5Hd+/ANMA3P0m4ALgcjNrAxqBZT7c/6VEZNBNmTKF8vJyknCMMS8vjylTphzSNlGeNfQU0OfVD+7+PeB7UdUgIgKQTqeZOXNm3GUMWbr7qIhIwikIREQSTkEgIpJwCgIRkYRTEIiIJJyCQEQk4RQEIiIJl5ggePWpCm6+5A/UVdbFXYqIyJCSmCBYe99r/P1tZ7L1ycq4SxERGVISEwSTZxUCUPmiegQiIpkSEwSlJ40CoOKV5pgrEREZWhITBJNOGY/RQcWOjrhLEREZUhITBOniPMZbNRW7EtNkEZF+SdS3Ymnubipq8uMuQ0RkSElWEBTVUlFXHHcZIiJDSrKCYHQjFU1j4y5DRGRISVYQTGilpmMMzU36ETQRkU6JCoLJpUFzKzfXxlyJiMjQkaggKJ2ZA0BF2d6YKxERGTqSFQTHFwFQ8WJ9zJWIiAwdyQqCk0cDUPFKU8yViIgMHYkKglGzJ5JPAxU7dLBYRKRTooLAigopzdpJxeupuEsRERkyEhUEAKV5e6jckxd3GSIiQ0bigmDyiHoq6kfGXYaIyJARWRCY2VQze8zMNpvZ82Z2RQ/rmJndaGZbzGyDmc2Lqp5OpeOaqWgeh+swgYgIEG2PoA34nLufAJwOfNrMTuy2zmJgVvhYDvwwwnoAKC2FZvLYs6sl6rcSETkqRBYE7r7T3deG03XAZqC022pLgds88AwwyswmRVUTZFxUtq46yrcRETlqDMoxAjObAZwKPNttUSmwI+N1OW8NC8xsuZmtNrPV1dVH9gVeOju4+2jFxj1HtB8RkeEi8iAwsyLgHuBKd9/ffXEPm7xl9N7db3b3+e4+v6Sk5IjqKZ0T3H204qUDR7QfEZHhItIgMLM0QQjc4e6/7GGVcmBqxuspQGWUNU2aF4w8VW7TMQIREYj2rCEDfgxsdvf/6mW1B4CLw7OHTgdq3X1nVDUB5IzMZ7xVURFp3IiIHD2yI9z3mcBFwEYzWxfO+xdgGoC73wSsBJYAW4AG4NII63nD5Lw9VOzWRWUiIhBhELj7U/R8DCBzHQc+HVUNvSkdUU9FrX6yUkQEEnhlMUBpSXBRmYiIJDUIJkO1l9BcpV8qExFJZhCEF5XtXBvpcWkRkaNCIoNgyuzgl8p2rNdFZSIiiQyCYxaMAWDr8/qlMhGRRAbB9PklpGhjy9Y+T2oSEUmERAZBOjeLGdnlbKnQtQQiIokMAoBji19ny+7RcZchIhK75AbBxHq2NEzSD9SISOIlNgiOeRvU+kj2bK+LuxQRkVglNgiOfXs+AFueqIi5EhGReCU3CE4Lfpdgy+p9MVciIhKvxAbBzLOnYnSw5fnmuEsREYlVYoMgb0wBU1OVbNke5Z24RUSGvsQGAcCxxVVsqRoRdxkiIrFKdhBMrGfLgUlxlyEiEqtEB8Exb3N2+zj2bdMBYxFJrkQHwbEnB7eY2PpEecyViIjEJ9lBEJ5C+tJz6hGISHIlOgiOP2cKKdooW98edykiIrFJdBDkjszj+JxX2bg1P+5SRERik+ggAJgzoYoNuyfHXYaISGwSHwRvP66Z7W1TqK2oj7sUEZFYJD4I5pwWDAuV/Xp7zJWIiMRDQbA4GBba8MTemCsREYlHZEFgZreaWZWZlfWyfJGZ1ZrZuvBxXVS19GXqGVMZSS0b13fE8fYiIrGL8o5rPwW+B9zWxzpPuvv5EdZwUJbK4u0jt7N++8g4yxARiU1kPQJ3XwXsiWr/A2neMftZV38sbQ0tcZciIjLo4j5GcIaZrTez35jZSb2tZGbLzWy1ma2urq4e8CIW/Fk2DRSy+VdbBnzfIiJDXb+CwMyuMLMRFvixma01s/ce4XuvBaa7+ynAd4H7elvR3W929/nuPr+kpOQI3/atFiwtBeCPKwc+ZEREhrr+9gj+xt33A+8FSoBLgRuO5I3dfb+714fTK4G0mY07kn0erlnvmsII9vPcc3G8u4hIvPobBBY+LwF+4u7rM+YdFjObaGYWTi8Ma6k5kn0erqyUcfrYl/jDNl1hLCLJ098gWGNmvyUIgofNrBjo83xLM7sLeBo43szKzeyTZnaZmV0WrnIBUGZm64EbgWXu7ofXjCP3zrn7KWuaxe4tuhOpiCRLf08f/SQwF3jF3RvMbAzB8FCv3P2jB1n+PYLTS4eEsz84Bn4HT936Eh/82sK4yxERGTT97RGcAbzo7vvM7BPAtUBtdGUNvgUXzSaPRp54uDHuUkREBlV/g+CHQIOZnQJ8HthO3xeKHXVyR+ZxxqjNPLF5QtyliIgMqv4GQVs4fr8U+I67fwcojq6seJw9p5Z1jcdR+9qw6uyIiPSpv0FQZ2b/DFwE/NrMUkA6urLi8c4PjMTJ4qmfvBx3KSIig6a/QXAh0ExwPcEuoBT4ZmRVxeT0vw6OEzzyq6a4SxERGTT9CoLwy/8OYKSZnQ80ufuwOkYAkD+2gEWj1vGb56fGXYqIyKDp7y0mPgI8B/wV8BHgWTO7IMrC4rJkwW5eaprO1nV1cZciIjIo+js09K/AAne/xN0vBhYCX4iurPgsvnQiAL/5ro4TiEgy9DcIsty9KuN1zSFse1Q59q9OZVbWVlY+lIq7FBGRQdHfL/OHzOxhM/trM/tr4NfAyujKilF2NouP38pjlcfRuL817mpERCLX34PF/wTcDMwBTgFudveroywsTucvK6aJfB76z+fjLkVEJHIW433eDsv8+fN99erVkb5HW30Tk0fU8c4Zr/GLV94R6XuJiAwGM1vj7vN7WtZnj8DM6sxsfw+POjPbH0258csuyuOjx67mgVdPZu/u9rjLERGJVJ9B4O7F7j6ih0exu48YrCLjcMkns2khlzv/7ZW4SxERidSwPPNnIMz79Bm8w9Zw0/8UcpSNnomIHBIFQW+Kirhs7rOU1UzmD79vjrsaEZHIKAj68NGvnMhI9nHTtTviLkVEJDIKgj4ULnknF4/5Nb94dhrVVRofEpHhSUHQFzMuuyKXFs/h1i/ooLGIDE8KgoM48Z/O493Zq/j2z0bT0BB3NSIiA09BcDD5+XzxwhfY1TyG//5aTdzViIgMOAVBP5x9wxIW8Tj/8e1smnUCkYgMMwqC/pgyhWvPeZrKAyP5/g36nQIRGV4UBP307h9cwLn2EF/6WjZVVQdfX0TkaKEg6Cc7bhbfuvBZGlqyufYfauMuR0RkwEQWBGZ2q5lVmVlZL8vNzG40sy1mtsHM5kVVy0CZ/e3L+Gz6Jn7082LWrtF1BSIyPETZI/gpcG4fyxcDs8LHcuCHEdYyMCZM4LobChjHbj67rEr3IBKRYSGyIHD3VcCePlZZCtzmgWeAUWY2Kap6BsrIKy/l34/7KX/YMoEffG1f3OWIiByxOI8RlAKZN/EpD+e9hZktN7PVZra6urp6UIrrVVYWf/Pgh1ic9TCfu66Aso3qFojI0S3OILAe5vX4reruN7v7fHefX1JSEnFZB2ezjuWnX3yVUR17+OiSfTQ2xl2RiMjhizMIyoGpGa+nAJUx1XLIxv/r3/HTk/+TsvLRXHFRXyNgIiJDW5xB8ABwcXj20OlArbvvjLGeQ5NKce4jn+OfC2/klnvGcMt3dCMiETk6RXn66F3A08DxZlZuZp80s8vM7LJwlZXAK8AW4BbgU1HVEpmJE/nyg/N4nz3Mp69K89vf6PeNReToY36UnQM5f/58X716ddxlvMne79/Juz5zIi9ln8iDD6V59zk9Hf4QEYmPma1x9/k9LdOVxQNg9Kc/xiOfvp9j2l7k/HNb+c3KoytcRSTZFAQDpOS71/G7i29jdlsZH3h/BytWxF2RiEj/KAgGihnjf/J1HrvoJyzseIZlH2nn9ts64q5KROSgFAQDKSuLkT/9Dg99cgVn+xNcdEkWX72+jQ7lgYgMYQqCgZaVRfGPvsXKr6zl49zOF76UzeI/28fOo+fEWBFJGAVBRPKu/Uf+555Cbhp1NU8+m8PJs5r57/+Gdp1hKiJDjIIgQvahv+Tvt17N2oWXc8KBP3LZZTBvTisbN8ZdmYhIFwVB1MaMYfZTP+LJLz/O3dmfoGpzDQvmtfHFa9upqYm7OBERBcHgSKexL1zLhWVf4E/v+hzntd3Pl/8txdTJbXzly06dfgZZRGKkIBhMxx/PxEdv557702yYeh5LWu7jui8as49p4ZvfhL174y5QRJJIQTDYzOADH+DtW+9jxc17+cPo8zm2+mk+/3mYWtrO1VfDpk3olFMRGTQKgrik0/B3f8efbb+LJ659lD/lncHSxrv5xjfgpJNg3jzn5z+Htra4CxWR4U5BELfiYvjKV5i7+1Hu+OZOtk46i+/wWZqf38qFF8Lkie0sXw4rVujUUxGJhu4+OtS0tMAvfkH77Xdx/8N5/MI/zIOppdS3F3DM25xlHzXOOw9OOw2yFOMi0k993X1UQTCUvfYa/OhHtN90C7+sPovvZ1/BU+1n0O4ppkxxLrnEWLoU5s8PDj2IiPRGQXC0a2uDRx6B229n372PsbJxEbdnX8rDbefQQRZFBe2ccmoWixYZ55wDZ5wBeXlxFy0iQ4mCYDg5cAAeegh+9SuqH13PrytO4TkWsjZ9Gqvb5tLuKfLynLPOCkLhnHNg3jxIpeIuXETipCAYrtzh1Vfhd7+D3/6W2gefZFXTAn6Xeh+/yzuPsgMzARg50lmwwHj722HSJDj2WDjlFJg+XQEhkhQKgqRoaAiGkH7/e3j8cV7fsIvf824e5108m30mWziGA21dY0Y5OXDMMXDccTBr1pufJ03ScQeR4URBkFR1dfDkk/Dss/DCC7BqFft2NbKJE9nEibw89nRezp/DSy0z2bJ3DM2tXd2D4mI4+WSYM6fr8fa3w8iRMbZHRA6bgkAC7lBRARs3wurVQUCUlcH27bSTRTlTeClvDi9POIvNOXPZ2H4iG14fz94DuW/sYto0mD076DFMm9YVGMceCxMmBK/VkxAZehQE0rf9+4P7WpSVBSGxaRO8+CLs2IEDFZSyIfsdbJjwHjZkzeXllulUNo6mcn/xW3Y1YkRw7KGkBMaPDx4TJ8LcuTB5MowbF4SIroEQGVwKAjk8u3dDVVUQDM89B+vXB9c27NgBBw7QgbGPUZRxMq+lj2XXuJPYXngiO2w6Ve1jqG4sorouj9r67DftNjs7CIqJE4NjEqWlwZDTqFFBr6KwMDj9tagoeIwcGaybkxPT5yAyDCgIZOBVV8O2bVBbG5y5tHlz0It4+eXgdcZNkvZTzDpOpXrEMVQXv43tObOo6hjHTibxUkMpO+uKaGjq+/Sl3NwgJKZNC57HjQvC5G1vC0KipCToaRQXB0Gis6FE3qyvIMjuaabIQZWUBI+etLbC66/Dzp2wcycjdu7k7MrK8PWzsPO+4AynHTugvh6AdrLYwxh2ZU+lIW8MrblF1KdHsz89lv1543nJjmcXE3ltxwQ2bRlBdUMhNXU5uPd8QCI/v6tHUVwclDpuXDAkZRYMWY0ZE/QycnKgoCB43Rkie/cGw1wTJwZnVhUVKVxk+Io0CMzsXOA7QAr4kbvf0G35IuB+4NVw1i/d/ctR1iSDIJ2GKVOCR1/cg6GmbdtIvfoqJbt2UbJ3LzQ2dj0aXoE9q6Hqp8EwVcbPurWRYjvTqWI8VYxnV+4M6vPGUZ89ivqcMcHDRlJ3YARV+0bxp7IiOiybdktRXZtDfcOhfbNnZQWhkpUVBEZnuHQ+xo4NAiM3NxjaKigIAik/v2s6Nze4xXheXhA8eXnBvM5hLx1olzhEFgRmlgK+D/wFUA780cwecPdN3VZ90t3Pj6oOGcLMgiPL06fDO9/Zv21aW984dpFdVcUxu3ZxzJ49sG9f8Gd87StBL2Pv3mC9mprg0dj41l2RTQs5tOYW01BYQnXBdJryR9NeUMyIEVCXM5ZKn8SrPoPG7GKayaGuo4j23AIarJDqAwXs3p3Nq1vTVNdkUVt75N/iOTlByBQWBuGROd3fR15eEFZ5eUH45OUFjxEjgudUKhi5M+s6HtP5zyHJFGWPYCGwxd1fATCzu4GlQPcgEOm/dDo4GDBp0qFt19DQFQo1NVBbS7qmhnR1Nezdy6i6OibX1UFdLdSVB2dS7aoLbulRVdWvH4ZoT+dRXzCeltximjyXhnHTaLQCGtvSNKSKaSweT3PeSLJSRmPuKPakSmhOF9Gcyqc5dwTk5tGclc+B5mwO7G/jwAE40JLmQGOK2v1Z7GzLpaHBaWjKoqHRaGgwWlsP83PMYBZ0znJyggDJzg4CJTs7eJ1KBb2fztfpdFe45IZnFrsH0+l0sJ+8vGC/ZkEPqLExCJ3OYBozJgimdDrYbvz4oLfVuV3nEN7Bnt2D/bt39b462yH9F2UQlAI7Ml6XA6f1sN4ZZrYeqAT+0d2f776CmS0HlgNMmzYtglJl2Ov8c3nq1EPftqMDmpqguTk4OF5dHTz27g1uG75/Pxw4QKq2lpGdQ1oAlZXgzcE3actO2PE7qGoMQmXfvjeOjxyWnBzIy6W1uIDG3FE05I6mIT2SBiukMVWEt7TSlDOChvyxNOWOpDl3BPtTo2nKKqA9K012fhofM5YDDXDggJFlTnNHNh35hbRl5dLQkqK9I4uO7DStnqaqCjwri/ZUDi1tKfbu7fpIIPhSbm4OfjOjqSn4CDpDoLN3Ul/ftU7U0ungY+/sTeXlBUGW+Uilup77mk6lgpo7/xboHMrLyema7u05JyfYvqOj67MoLAxCsLAw2L0WM9gAAArHSURBVGdnkKXTwev29mCbESO66nAPlpWUHHzE9XBEGQQ9dTS7n6K0Fpju7vVmtgS4D5j1lo3cbwZuhuCsoYEuVKRPWVldQTJ6NMyYMTD7bW7uCoSqqiBQ9uwJvgmKioJvr8YwONraoLw8qKGpKVi/pYV0SwvpxkZGNDYGodTWBvXlwbdMcw3sXw+1+4MA27+/a39HorO70N4efNu3tATfZvn5wcGSMX18O2Zn09oKe/JLqcsfT2sqj0YrCI7ZdBTQlC7G8wvwgkI6UmncjI5UTvBsKdxSwXNWFh2kyMrOIis76LY0tqRobE3R3JqiqTVFB1k0NGVxoDGLphajvcNobQ16Ue3tXY+2tuCfoqGh64u4c37ndCoVfFG7B81taQm2yXweiN7ZwVx9Ndxww8HXO1RRBkE5kPnn1xSCv/rf4O77M6ZXmtkPzGycu++OsC6RoaHznNgJE4JTkwZLe3sQCtu3BzV0ju+0twdndjU2dn0D1tYGrwsKgmGyzuMu7e1d4y+d29fXB8HW/RuyuTm43Un4bZlOpZiw91km1NZ2fYNmZ3f9eRyl7t2Cnh4QfOsDjB4ZhGpTU9D++vqgxmmlwfhW5+eXm0tHTh6t2fk0ZxfSksqnJbuA5twRQZB4G1npFO2WzYHUCGpb8mmggOysDlLZwVhXCzmk2xpJWQdt7UZdRyFtuYW05+RhWVmk086sOQXAuIH/WAZ8j13+CMwys5lABbAM+FjmCmY2EXjd3d3MFhL8dGbNW/YkIgMnlQp6NqNHv3XZrLd0yKPX0dF1pLq+Pgif2trgy7e9nTf9Cd/ZO4rq0flnfWtrUFddXVBT5/nFxcVBt6CyMrheprn5jUdWczO5LS3kNjdH1z24+mo4e+C7BJEFgbu3mdlngIcJTh+91d2fN7PLwuU3ARcAl5tZG9AILPOj7Qo3ETkymUd2i4uDRxQD4YOpo6OrJ5RKBW1sawvCrakpmG5sDOZ3dHSNTxUVBfPcgwDavz8Ys+o8sHLCCZGUqyuLRUQSoK8ri3WSlYhIwikIREQSTkEgIpJwCgIRkYRTEIiIJJyCQEQk4RQEIiIJpyAQEUk4BYGISMIpCEREEk5BICKScAoCEZGEUxCIiCScgkBEJOEUBCIiCacgEBFJOAWBiEjCKQhERBJOQSAiknAKAhGRhFMQiIgknIJARCThFAQiIgmnIBARSTgFgYhIwkUaBGZ2rpm9aGZbzOyaHpabmd0YLt9gZvOirEdERN4qsiAwsxTwfWAxcCLwUTM7sdtqi4FZ4WM58MOo6hERkZ5F2SNYCGxx91fcvQW4G1jabZ2lwG0eeAYYZWaTIqxJRES6iTIISoEdGa/Lw3mHug5mttzMVpvZ6urq6gEvVEQkyaIMAuthnh/GOrj7ze4+393nl5SUDEhxIiISiDIIyoGpGa+nAJWHsY6IiEQoyiD4IzDLzGaaWQ6wDHig2zoPABeHZw+dDtS6+84IaxIRkW6yo9qxu7eZ2WeAh4EUcKu7P29ml4XLbwJWAkuALUADcGlU9YiISM8iCwIAd19J8GWfOe+mjGkHPh1lDSIi0jddWSwiknAKAhGRhFMQiIgknIJARCThFAQiIgmnIBARSTgFgYhIwikIREQSzoJruo4eZlYNbD/MzccBuwewnKOB2pwManMyHEmbp7t7j3ftPOqC4EiY2Wp3nx93HYNJbU4GtTkZomqzhoZERBJOQSAiknBJC4Kb4y4gBmpzMqjNyRBJmxN1jEBERN4qaT0CERHpRkEgIpJwiQkCMzvXzF40sy1mdk3c9RwJM7vVzKrMrCxj3hgze8TMXg6fR2cs++ew3S+a2fsy5r/DzDaGy240MxvstvSHmU01s8fMbLOZPW9mV4Tzh3Ob88zsOTNbH7b5S+H8YdvmTmaWMrM/mdmD4eth3WYz2xbWus7MVofzBrfN7j7sHwQ/lbkVeBuQA6wHToy7riNoz9nAPKAsY943gGvC6WuAr4fTJ4btzQVmhp9DKlz2HHAGYMBvgMVxt62X9k4C5oXTxcBLYbuGc5sNKAqn08CzwOnDuc0Zbf9/wJ3Ag8P9v+2w1m3AuG7zBrXNSekRLAS2uPsr7t4C3A0sjbmmw+buq4A93WYvBX4WTv8M+GDG/LvdvdndXyX4feiFZjYJGOHuT3vwX9FtGdsMKe6+093XhtN1wGaglOHdZnf3+vBlOnw4w7jNAGY2BTgP+FHG7GHd5l4MapuTEgSlwI6M1+XhvOFkgrvvhOCLExgfzu+t7aXhdPf5Q5qZzQBOJfgLeVi3ORwiWQdUAY+4+7BvM/Bt4PNAR8a84d5mB35rZmvMbHk4b1DbHOmP1w8hPY2VJeW82d7aftR9JmZWBNwDXOnu+/sYAh0WbXb3dmCumY0C7jWzk/tY/ahvs5mdD1S5+xozW9SfTXqYd1S1OXSmu1ea2XjgETN7oY91I2lzUnoE5cDUjNdTgMqYaonK62H3kPC5KpzfW9vLw+nu84ckM0sThMAd7v7LcPawbnMnd98HPA6cy/Bu85nAB8xsG8Hw7bvN7HaGd5tx98rwuQq4l2Aoe1DbnJQg+CMwy8xmmlkOsAx4IOaaBtoDwCXh9CXA/Rnzl5lZrpnNBGYBz4XdzTozOz08u+DijG2GlLC+HwOb3f2/MhYN5zaXhD0BzCwfeA/wAsO4ze7+z+4+xd1nEPw/+nt3/wTDuM1mVmhmxZ3TwHuBMga7zXEfMR+sB7CE4GyTrcC/xl3PEbblLmAn0Erwl8AngbHA74CXw+cxGev/a9juF8k4kwCYH/5HtxX4HuGV5kPtAZxF0M3dAKwLH0uGeZvnAH8K21wGXBfOH7Zt7tb+RXSdNTRs20xwJuP68PF853fTYLdZt5gQEUm4pAwNiYhILxQEIiIJpyAQEUk4BYGISMIpCEREEk5BIIljZv8XPs8ws48N8L7/paf3EhnKdPqoJFZ4G4N/dPfzD2GblAe3fuhteb27Fw1EfSKDRT0CSRwz67yr5w3An4f3gb8qvMnbN83sj2a2wcz+Plx/kQW/h3AnsDGcd194k7DnO28UZmY3APnh/u7IfC8LfNPMysJ7xl+Yse/HzWyFmb1gZnd03kfezG4ws01hLf8xmJ+RJEtSbjon0pNryOgRhF/ote6+wMxygT+Y2W/DdRcCJ3tw61+Av3H3PeHtH/5oZve4+zVm9hl3n9vDe30ImAucAowLt1kVLjsVOIng3jB/AM40s03AXwKz3d07bzchEgX1CES6vBe4OLz187MEl/nPCpc9lxECAJ81s/XAMwQ3AZtF384C7nL3dnd/HXgCWJCx73J37yC4fcYMYD/QBPzIzD4ENBxx60R6oSAQ6WLAP7j73PAx0907ewQH3lgpOLbwHuAMdz+F4J5Aef3Yd2+aM6bbgWx3byPohdxD8AMjDx1SS0QOgYJAkqyO4KcvOz0MXB7e8hozOy68I2R3I4G97t5gZrMJfkKyU2vn9t2sAi4Mj0OUEPzc6HO9FRb+9sJId18JXEkwrCQSCR0jkCTbALSFQzw/Bb5DMCyzNjxgW03PP/f3EHCZmW0guAPkMxnLbgY2mNlad/94xvx7CX5Pdj3BnVQ/7+67wiDpSTFwv5nlEfQmrjq8JoocnE4fFRFJOA0NiYgknIJARCThFAQiIgmnIBARSTgFgYhIwikIREQSTkEgIpJw/x+78X14gPWL8QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss_history_valid = compute_loss_history(w_history,valid_X[0:1000,:],valid_y[0:1000],reg)\n",
    "\n",
    "plt.plot(loss_history, color='r')\n",
    "plt.plot(loss_history_valid, color='b') \n",
    "plt.ylim(0,5)\n",
    "plt.xlabel('iterations')\n",
    "plt.ylabel('loss')\n",
    "plt.title('iterative learning curve')\n",
    "plt.legend(['train', 'valid'])\n",
    "plt.ylim(-0.2,3)\n",
    "plt.show()    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型参数矩阵 𝑊 是一个 𝑛×𝐶 矩阵，其每一列对应一个类似逻辑回归的分类器，该列的权值用于从数据中提取能判断数据中和该类相关的特征。对于MNIST图像分类的模型参数 𝑊 ，可将其某一列（对应某个分类）的784大小的权重参数以图像形式显示出来。如下面的代码显示第0列（对应数字0的那个类）的权重参数，将这列向量转化为 28×28 大小的图像矩阵："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAATvklEQVR4nO3dS2xc130G8O+bF9+USEnU27Fku0XdIlUCwm3hIHARNHC8sbNIES0CF3CrLGIgQbOo4S7ipVE0DrIoAiixEaVIHQRIDGthtBGMFEaAIjDtynpEjqUqqk1LJiVLFB8jivP4d8HrgrF5/4eeO3dmpPP9AILknLlzDy/nmzvk/55zaGYQkdtfodsdEJHOUNhFIqGwi0RCYReJhMIuEolSR3c2Omjlic2d3KVIVGqzc6jPV7leW6awk3wQwHcBFAH8wMye9u5fntiMu5/52yy7FBHHub//QWpby2/jSRYB/AuALwC4F8BBkve2+ngikq8sf7PfB+CcmZ03sxUAPwHwcHu6JSLtliXsuwG8s+b76eS230PyEMkpklON+WqG3YlIFlnCvt4/AT5y7a2ZHTazSTObLI4OZtidiGSRJezTAPau+X4PgIvZuiMieckS9lcB3ENyH8kKgC8DONqebolIu7VcejOzOsnHAfwHVktvz5nZ6bb1TETaKlOd3cxeAvBSm/oiIjnS5bIikVDYRSKhsItEQmEXiYTCLhIJhV0kEh0dzy7rC83va7bu8OSNPXZg29DkwuHtW+8bmW1m49D2dLpWKDSz7TvT1t2hM7tIJBR2kUgo7CKRUNhFIqGwi0RCYReJhEpvG+QVebKUnwCg0fBfc5vN1stfzcBjB8t+jYylN/fAZSsLshAovRXT24tFv/RWLPntpVLD33eoLOi25kNndpFIKOwikVDYRSKhsItEQmEXiYTCLhIJhV0kEqqzJ5oZhnKG6uDBOnqoFh6qZTs13dBQzr6KXy8uF/32wb4Vt325lv4UqzWK7rY1Z1sAaNQDx9U5bqFjGvqd1ut+30N1fO/3klcNXmd2kUgo7CKRUNhFIqGwi0RCYReJhMIuEgmFXSQS0dTZQ+O2s9TKQzXXZi1QRw/UixkYW10ZqKe2jQzedLe9a+yK275/0G8fKy+57Z7r9UG3/UJ1i9t+qTrqtl9eHEptW1nxn/qhax9qgfZGYB6ASiW9LXRtRKt1+ExhJ3kBwAKABoC6mU1meTwRyU87zux/aWb+y7+IdJ3+ZheJRNawG4BfkHyN5KH17kDyEMkpklON+WrG3YlIq7K+jb/fzC6SnABwjOSbZvbK2juY2WEAhwFg4O5d2Rb3EpGWZTqzm9nF5PMsgBcA3NeOTolI+7UcdpJDJEc++BrA5wGcalfHRKS9sryN3w7gBa6ui1sC8G9m9u9t6VUOwuOX/dc9r5beXPHr7LYSeE0NzY/e59+h5NThh/v8OvtAsea218z/2fqZXuMHgP19M6ltZfhj5ff1+XX2k3173PYL5fTtry77Nf7rN/rd9qVqn9veqAXG6jtPx0oltBR1a38Ntxx2MzsP4E9b3V5EOkulN5FIKOwikVDYRSKhsItEQmEXicRtM8Q1+xDWwFTSTrvVA4MOA/tmv1+CCk1L7E33XAiUaa7d9EtQJfr7Hiv5l0A3LP18srt03d12qOCXDatNv/x1vTaQ2rbS9EtjoeNWCwxrXl5yxrDCH0KbZepwj87sIpFQ2EUiobCLREJhF4mEwi4SCYVdJBIKu0gkbqk6u1ddDNUmw8smB4apetuH6qKVwNTAgamiyxV/GGnJqcOXC34Nv7/kD3EtBOrs1aZfT55r+HV8z2hh2W0fKd5w2yf6FlLbbjTK7rbDZb/GXw8MiX4vMFV1sJaeA53ZRSKhsItEQmEXiYTCLhIJhV0kEgq7SCQUdpFI3FJ1dk94qujAePXQEOFC+h0K/X4dnIGX1FAdvb/i18KHKiupbRMD6bVmANhWWXTb52p+nfzE9d1u+5sL21PbJvr9fR8Yfttt31L0t99ZmUttKyJwbUPg+oSRkl+HX1j2x9pXnamoW50qOkRndpFIKOwikVDYRSKhsItEQmEXiYTCLhIJhV0kErdRnT3b9nTq6ABQcNorfX4dPMQbjw4A/WW/Dr9rKH3+9dFAPfjd5c1u+9mrW932qzOjbjtvps8TUN7qj0ef3+svm/zpUb8Ov7U0n9q2d+iqu+02Z1sAeLOyy22/sDjutl+sdT56wTM7yedIzpI8tea2cZLHSJ5NPo/l200RyWojb+N/CODBD932BICXzeweAC8n34tIDwuG3cxeAfDh9zwPAziSfH0EwCNt7peItFmr/6DbbmaXACD5PJF2R5KHSE6RnGrM++uCiUh+cv9vvJkdNrNJM5ssjrY++aCIZNNq2GdI7gSA5PNs+7okInloNexHATyafP0ogBfb0x0RyUuw2EfyeQAPANhKchrAtwA8DeCnJB8D8DaAL+XZybYIjHcPrcc9NJherx7sSx9PDgC1wJz0oX1vGfD/1zFUSt//hSW/3vvWe9vcdp4bctvH/VI3Sk4pfXHPsLvtf9X2ue19d/vXH/zZpvR55/+4fNHd9u6yP569gnfc9v8c+EO3fXYh/WfPazx7MOxmdjCl6XNt7ouI5EiXy4pEQmEXiYTCLhIJhV0kEgq7SCRumyGuzLgCbv+AXz7bPpI+JXOl6Jdplmr+ssYjgeWB7xx+322/XhtIbXt7zh/CWnjLL61t+29/+O3Q2/50zqylH5vK0iZ325lRf4jric3+MNPtfenDVD/Z55fONhX839mukv9zj1f8cmmpkH5cmzkt56wzu0gkFHaRSCjsIpFQ2EUiobCLREJhF4mEwi4SiVuqzp6l+hiaKnp80J/WeO9Q+vK/A0W/Rr9Q9+vFO/vSp4IGgE3eOFEAZ+Z2pLYtXfBr2dtP+8dl5Dd+jR8zl/12Z73q/rH06wMAoDzvt8/N+zMfTW9Jv8ZgfsT/nVSbS257IzAKtRRY8tkdxqo6u4hkobCLREJhF4mEwi4SCYVdJBIKu0gkFHaRSNxSdfYsioFlkfcMp9fRAWDfwJXUtk0lf+xy0/zX1F3la277OzV/OujpufRa+vDv/H2Pnk8fpw8gWEdvXPeXNi4MptfC6wP+06/pDylHKTDdc5npv/OrDX8a69/V/Z9ryfrc9tCYdMuplu7RmV0kEgq7SCQUdpFIKOwikVDYRSKhsItEQmEXicRtU2e3wPjicsmvye7s98eU76qk18J3lPxtRwr+ePR++ksPn1n250evzqTP/b77Xf/nLl666rbX5/yfLTRhf2Fia2pbdcJ/+i1v94/L7hF/zHlfMX370LULZfrHrWb+MtxXV/z5+POaG94TPLOTfI7kLMlTa257iuS7JI8nHw/l200RyWojb+N/CODBdW7/jpkdSD5eam+3RKTdgmE3s1cA+O/1RKTnZfkH3eMkTyRv88fS7kTyEMkpklONef8achHJT6th/x6AuwAcAHAJwLfT7mhmh81s0swmi6P+BIEikp+Wwm5mM2bWMLMmgO8DuK+93RKRdmsp7CR3rvn2iwBOpd1XRHpDsM5O8nkADwDYSnIawLcAPEDyAAADcAHAV3Ps44aExgcXvHm6AQwX/TXSvVr6eNFfq3tH4LEXmn7N9krNH3tdXErfvnTDr1WH6uTF0VF/8xG/b9U/2JbaNr/f3/fQDr+Ovms4cA2A40ptxG0PzUFwve7PaX9l2a+ze8/GvCrwwbCb2cF1bn42h76ISI50uaxIJBR2kUgo7CKRUNhFIqGwi0Tilhri6pUrQqU3d4lcAEVn2mEAKDvDUIcCQ1RDr6iNjMUWK6f/bMtjflmvb7c/1LOwyS+tLe/y2698Mn0+6JW7/KG/+zf50zmXAr+zxXr6vm82/Kf+jUbZbZ+upi8HDQDXb/pLQnvP19BztVU6s4tEQmEXiYTCLhIJhV0kEgq7SCQUdpFIKOwikbil6uyeZtOvVdeb/utaaBjp+84Sv5sDU0UPmj8tcWha4U0l//FtfCW1bWmXX+8FArMHmd++uMc/rkv70q9B2DrmD2ENXfuwWPeXTV5ppF9j0O9MMw0Al2/6z4fLVX8Ia/Wmv950XrV0j87sIpFQ2EUiobCLREJhF4mEwi4SCYVdJBIKu0gkbps6e2jJ5nrdH9f93rI/ZfJ035bUtn7W/J2X5tzmojtSH7ij8r7bvm/XldS287UJd9vlLf64bSv6fWts8n/2kW3p02wP9/lTbNcCU2xnGTNeKfrXPizX/WiE6ughhYJ/DUEedGYXiYTCLhIJhV0kEgq7SCQUdpFIKOwikVDYRSJx29TZAysPB11b9sdtn6uk16tD877PNfzHvrOSXidfbb/stn9227nUttDc6hfH/esLQtcnjA34tfJKKX3cuDfeHAAWl/3x6vWGf64qFNKvEagGxpPXan40Qtd1FIudr6OHBM/sJPeS/CXJMyRPk/x6cvs4yWMkzyafx/Lvroi0aiNv4+sAvmlmfwTgzwF8jeS9AJ4A8LKZ3QPg5eR7EelRwbCb2SUzez35egHAGQC7ATwM4EhytyMAHsmrkyKS3cf6Bx3JOwF8CsCvAWw3s0vA6gsCgHX/qCV5iOQUyanGfDVbb0WkZRsOO8lhAD8D8A0z81fcW8PMDpvZpJlNFkcDkxuKSG42FHaSZawG/cdm9vPk5hmSO5P2nQBm8+miiLRDsPRGkgCeBXDGzJ5Z03QUwKMAnk4+v5hLDzcoa6ljqeYPWbxyM33q4ErBn5a4af5r6uai/+fNHaVrbvtnhn/rtnvO9O9w26vOssdAeBrsxZX08tm16oC77c0V/+nZCJQFvemaMy/xXfKfb17ZD0DGRbpbs5E6+/0AvgLgJMnjyW1PYjXkPyX5GIC3AXwpny6KSDsEw25mv0L6C9Hn2tsdEcmLLpcViYTCLhIJhV0kEgq7SCQUdpFI3FJDXL3aZKiuWa/7r2vVFX9K5fmV9GmL+4v+dMqhJZdDQ2C3FNOnYwaAO0vXU9tGRt9wt/2LobNu+xvLd7jt56r+VNXnG1vddk9oeG0z8DuFs4w3A3XyUtlvD13X0Y0lmUN0ZheJhMIuEgmFXSQSCrtIJBR2kUgo7CKRUNhFInFL1dk9oSVwi37JFiuBJXpnFkZS25broRq9P2772pBfZ7/S70/3/AlnKuoy/bH2l+v+Y8+u+O3vLPmTCr83n37cblT9qaIby4FfWgCd5aZD12VkraN3Y7x6iM7sIpFQ2EUiobCLREJhF4mEwi4SCYVdJBIKu0gkbps6e6iuGaqbNpv+695NZwnfizf8WvS7zc1u+ynsdNtDvJ+9WGpkeuzgmPKaf9zMG3MeGvIdOBUVyv7PVnDq7KHjEpz3vQfHq4fozC4SCYVdJBIKu0gkFHaRSCjsIpFQ2EUiobCLRGIj67PvBfAjADsANAEcNrPvknwKwN8BuJzc9UkzeymvjmYVqsOHxsN7r4uhmmwj8NCh+c/dWjXg1qvrTX+sPbLWiwNd88aUM3Dtg1cnB7LVyoPj0W/BOnrIRi6qqQP4ppm9TnIEwGskjyVt3zGzf86veyLSLhtZn/0SgEvJ1wskzwDYnXfHRKS9Ptbf7CTvBPApAL9Obnqc5AmSz5Fcd34ikodITpGcasxXM3VWRFq34bCTHAbwMwDfMLN5AN8DcBeAA1g98397ve3M7LCZTZrZZHHUn2tNRPKzobCTLGM16D82s58DgJnNmFnDzJoAvg/gvvy6KSJZBcNOkgCeBXDGzJ5Zc/vaoVpfBHCq/d0TkXbZyH/j7wfwFQAnSR5PbnsSwEGSB7Ba+LkA4Ku59LBDspTmyuVs0xJbxd+7WX4TE1ugwhTad5YSVdZhpKHtszx2cPtMW3fHRv4b/yus/7P1bE1dRD5KV9CJREJhF4mEwi4SCYVdJBIKu0gkFHaRSNw2U0nnzaurBuvBgaGaecq65zzr7FndirXubtKZXSQSCrtIJBR2kUgo7CKRUNhFIqGwi0RCYReJBC00oLmdOyMvA/jfNTdtBXClYx34eHq1b73aL0B9a1U7+/YJM9u2XkNHw/6RnZNTZjbZtQ44erVvvdovQH1rVaf6prfxIpFQ2EUi0e2wH+7y/j292rde7RegvrWqI33r6t/sItI53T6zi0iHKOwikehK2Ek+SPK3JM+RfKIbfUhD8gLJkySPk5zqcl+eIzlL8tSa28ZJHiN5Nvm87hp7XerbUyTfTY7dcZIPdalve0n+kuQZkqdJfj25vavHzulXR45bx/9mJ1kE8BaAvwIwDeBVAAfN7Dcd7UgKkhcATJpZ1y/AIPlZAIsAfmRmf5Lc9k8ArprZ08kL5ZiZ/UOP9O0pAIvdXsY7Wa1o59plxgE8AuBv0MVj5/Trr9GB49aNM/t9AM6Z2XkzWwHwEwAPd6EfPc/MXgFw9UM3PwzgSPL1Eaw+WToupW89wcwumdnrydcLAD5YZryrx87pV0d0I+y7Abyz5vtp9NZ67wbgFyRfI3mo251Zx3YzuwSsPnkATHS5Px8WXMa7kz60zHjPHLtWlj/PqhthX2/qsF6q/91vZp8G8AUAX0versrGbGgZ705ZZ5nxntDq8udZdSPs0wD2rvl+D4CLXejHuszsYvJ5FsAL6L2lqGc+WEE3+Tzb5f78v15axnu9ZcbRA8eum8ufdyPsrwK4h+Q+khUAXwZwtAv9+AiSQ8k/TkByCMDn0XtLUR8F8Gjy9aMAXuxiX35PryzjnbbMOLp87Lq+/LmZdfwDwENY/Y/8/wD4x270IaVf+wG8kXyc7nbfADyP1bd1Nay+I3oMwBYALwM4m3we76G+/SuAkwBOYDVYO7vUt89g9U/DEwCOJx8PdfvYOf3qyHHT5bIikdAVdCKRUNhFIqGwi0RCYReJhMIuEgmFXSQSCrtIJP4P3qIl69P5KLgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "c = 0\n",
    "plt.imshow(w[1:,c].reshape((28,28)))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fasion MNIST训练集的softmax回归"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784) (60000,)\n",
      "uint8 uint8\n"
     ]
    }
   ],
   "source": [
    "import mnist_reader\n",
    "X_train, y_train = mnist_reader.load_mnist('data/fashion', kind='train')\n",
    "X_test, y_test = mnist_reader.load_mnist('data/fashion', kind='t10k')\n",
    "print(X_train.shape,y_train.shape)\n",
    "print(X_train.dtype,y_train.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "显示其中的一些图像："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU4AAAD7CAYAAAAFI30bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOy9W2xk2XUe/O26nLrfSBbv7HvPtGZGox5ppJFHmnGsQLIuMRw9xIoCB7EdwC8JYBt5sJyX/G+/ngL8QAIYAmJINow4AiRAgS3DNmRFkUZSe6ZHo+mZ7pkedjf7xmsVWfd71fkf2N/mqs1TbF6KxSL7fABRZNWpU4dn1V57Xb61lrJtGy5cuHDhYvfwHPUFuHDhwsVxg6s4Xbhw4WKPcBWnCxcuXOwRruJ04cKFiz3CVZwuXLhwsUe4itOFCxcu9ogDKU6l1OeVUu8rpeaVUl/r10W5OFq4cj25cGXbH6j98jiVUl4ANwF8FsADAK8D+Kpt29f7d3kuBg1XricXrmz7B98B3vsJAPO2bd8GAKXUXwH4TQA9haCUetLZ9hnbttNHfRGPwdDI1ePxwO/3w+fzYXR0FIFAANVqFbVaDe12G41GA7Ztg5s/H30+H/x+P/x+P8LhMJRSaDab6HQ6KBaLqFQqsG0bnU6nX5d6HOQK7FG2/Zarx+PRMo1GowCAUqmkZbNbeXg8HliWpc/l8XhQrVZRr9fR54KennI9iOKcAXBf/P0AwEsHON+TgLtHfQG7wMDl6vF4oJSC1+vVi8vr9SIcDmNychLpdBpf+cpXcO7cOVy/fh3z8/PI5XJYXFxEq9VCu91Gp9NBq9UCAIyOjiKdTiOdTuP555+Hz+dDJpNBuVzGT3/6U/zyl79EvV5HtVpFp9NBu92Gbdv6XFIZ7xLHQa7AEa/ZcDiMeDyO6elpvPLKKwCAK1euYHl5GaVSCeVyuet4ykAp1fV8LBbD7OwswuEw0uk0LMvCu+++i/n5ebRaLTQajX5dck+5HkRxKofntn3blFK/D+D3D/A5LgaLgcrVsiyMjY0hFArh3LlzGBsbw+TkJObm5uDz+RAMBhEMBnHx4kUkEglEo1E8++yzWuEB0JYKFV4gEEAgENCK2LZtTExMoNVq4ZlnnkG9XketVkO5XEalUsG9e/dQLBbxzjvvYHFxEYVCAfl8/qD/2jDisbI9jPXq9/vh9XoxOTmJ8+fP49SpU3jppZcQCATw9NNPo1wua1nQE7BtW8uX7+eGGgqFMD4+Dtu2sbKyglKphHa7jXA4jEwmgzt37uj3HhYOojgfAJgTf88CWDQPsm37GwC+Abiu+jHBQOXq9XoRj8eRSCRw4cIFnDlzBhcvXsRzzz0HpRQ6nQ6UUggEAvB4PBgdHYXX60UgEEAkEtlmjUhUKhUsLy+j1Wpp6yWdTiOVSqFWq6FYLCKfz+PatWvIZDIoFAool8totVooFAr9dvuGAY+V7WGsV4/HA5/Ph2QyienpaczNzeHMmTOIRCI4c+YMOp0OyuWy9gAoL3oQwWAQPp8PPp8PXq8XwWAQqVQK9Xodv/jFL7C6uopyuayV7sLCQj8ue0ccRHG+DuCiUuosgIcA/jWAf9OXq3JxlDhUuXq9Xni9XkxPT+O5555DLBbD6dOnEYlEMDc3h1QqhZGREW0xcPG0Wi0opbRb7/P5UCwW9aJUSmmXHdi0PhuNhrZiqARzuRzq9Tra7TaazSba7TbGxsYQDofx6quv4plnnsHy8jIePnyI1dVVvP3226jVav36948aR7JmGf5oNBpoNBpYXV3Fz3/+c0SjUZw9exbxeByhUAjhcLhLnoSMYVOh0r1fWFjA6uoqCoXCnuKkB8W+Fadt2y2l1H8E8HcAvAD+zLbtd/t2ZS6OBIctV5/PB8uy8PTTT+MrX/kKRkZGcOrUKQSDQR3fbDabWqkxcSBjj7Ztb4uFKqXQaDS0tWImjIh6vQ5gU4H7fD54PB5MTExAKYWLFy9CKYUHDx7g7t27uHbtGj744IMToziPas1y82OIZGlpCffv30c8Hkc0GkUkEkEsFkM0GkUwGEQsFtOJH2DTc2g0Gmg2m2g0GlhfX8eDBw+QzWZx8+ZNrKyswLIs+P3+gXkJB7E4Ydv29wF8v0/XMjDQagkEAgiFQmg2myiVSj13K6XUSXTbeuKw5KqUwtTUFGZnZ3HhwgWMjIzo7Kq0Ftvttv6b8S7z/vO5TqeDRqMBj8eDVqvV83gT8tz8PvD3YDCI0dFRfX21Wg31en1g1sxh4qjXrMfj0e54rVbDysqK9kCYKa/X6/B6vfp+1+t1NJtNlMtllEolrK2tYXFxEblcDu12G5Zl6bDOYcc2iQMpzuMKUly4iDc2NvDee+9pa0SC7mGf6StPJDweD1599VV86UtfQiqVwtTUFACg0WigVqtphaeU0orMXAh8TR5TrVb1+0z0ioFS4ZKqJBEKhXDx4kWUSiXMzMzA6/ViZWUF1Wq1T3fiyQSZE8Dm5lgqlXD16lW89957+MQnPqHpZtwIfb5N9USqUSaT0WGU1157DbVaDdFoFPF4XB8z9K76cYBpKTIzR5cgkUggHo+j1WohGAwC2HLtaPW46A/ooicSCUxMTCAUCsHv92urT8anaAEylsnnJKRClHKSz/McEtLd72WdSm8kkUigUqkgm83250a4ALAlp3K5jGazifX1dWQyGQQCAdRqNXi93i5XvV6vI5vNIpPJIJfLaUUZjUZ1THSQXuGJVZxy0di2Db/fj1QqhVAohOeffx4zMzOaKM0sa61W0zGytbU1rK+vO1o9Tp9BOO14XPRPqsXq8/kwOTmJeDyOqakpjI6O6kyqtDJpjTCD6vf7EYlE9CKiQgXQ5QFI61Ny/+Q5qSQZP63X6zo84ySXTqeDZDKJl156CUtLS9jY2ECxWBzE7TqxoHuulEI4HNbfgUKhgJ/97Ge4du0aQqEQ4vG4lr3H48HGxgYqlQqKxSIKhYJOFIbDYR2nlt+NQeDEKk6g2+Jk7CoajWJ6ehrnzp3Ti6jT6WB0dBS1Wk0nJZixBR6v8EwrR+58gxTmsMLj8SASiSAejyMSiSAYDHZtUl6v19FSJO3I6/XqOBYXCJNFPJ6xM5M0TYuVCpKLjL8DcLQ6yQedmprSv7s4ONrttt4UWclVq9V0uCUcDiOZTMLn8yEej8Pj8SCbzWqeZ7VahWVZekMdtMIkTqziNONglmVhbm4OY2NjuHDhAs6fP49qtYpKpYJoNKpL+CiEubk55HI5TahtNpuo1WpdStTM3NK1lFbRThbrkwLGk6empjAyMqIXjtfr7YpNSgu+3W5jZWUFV69e1TSWdrutEzV+vx+BQACdTkfTiwhpxVIeMkwTCoUwMjKC2dlZANDxVdNLCYfDuHTpks7+ujgYZHgE2JR3MBjUXgGTc5VKBR6PR7MZmPQLBAI6gURL04m2NAicaMXJpA6wWaEyNTWlq1JOnTqlTf9wOKz5YRRMqVRCrVbD/Pw8MpmMpkSY1qcpLPIU+Xyr1dJ0jCcVXq8X4+PjmJubQzwe1/eI96lXUiibzeLKlSvI5/PI5XKatF4sFhEKhZBMJtFqtZDL5fQ9lllbfga9DZ/Ph5mZGYyPj+PSpUs4ffo0vF6vTgry8/m9CYfDOHv2LAKBAMLh8FHewhMBM66slNKJWoJhFEk/syxLb7aWZelzyE1u0DixipOLjzfV6/UikUhgdHRUl+5JazCRSKBer+tgNa2TqakpXL58GeVyGWtra5pUTQHXajXdtMCyLCSTSQQCAZRKJVQqFVQqFeRyOR1fexLh8/m04ozFYtvcaRlO8Xg8Wm7lchkLCwvI5/N6Q2NSqV6vY2lpSVceeb3ebRlZno/KtNFoYHFxEdlsFrFYDO12uysRxWOBLZcyGAwiHA4jGo0iFouhXq/3sxb6iYaTxyYbu9BIkRvgTnDKORwWTqziNC1Dy7IwPT2NU6dOYWJiAqOjo9pdZMUClWMul8Pk5CRGR0cxOjqKD33oQ6hWq1hYWECpVMLS0hLy+bzOBMZisa4SskgkggcPHmBpaQmZTKaLzP0kwrIsXLx4EZcvX0YymeziT0rXjaR0xiPX19fx+uuvo1Ao4Omnn9abXiQSwZ07d3Dz5k3EYjG88soriMfjWF5eRj6f16V5UmmWSiXU63XcuXMH2WwWkUgEzWazy5Xn+1jl4vP5EIvFUKvVkE6nMT4+jmw26yrOfULGqAlaoVSKTAw6QSpOMxxGGbqKs0+g1UCLgfQFLly6AKFQCACQTCbh8XgQDoe7XEkmKtrtdle8y+fzIRwO64w94zXRaBQTExOwbRuZTEbXPz+JYAaU2VLTNZchFQBdll+z2YRt25pzG4lEEAqF0Gg0cPfuXSQSCZw6dQqJRAIANI0oFovpc7fbbWxsbKBaraJQKOgKI0l/kp8vKVI8jt+hJ1WG/YBUnI9TcDtZlzK5Z/J6B4UTrziTySTOnDmD06dP4/z585iZmYFlWTrREwgEuhIN6XRa88ry+TzK5TJyuZx230OhEKanpzXFic0nvF4v2u02VldXkcvlcPr0abzwwgu4deuWLhVbXl4+6ttxJKCrfurUKR07lu65rNrh8UwcKKUQj8fxW7/1W/jUpz6l3fj3338fp0+fRjKZxOc//3kkk0lcu3YNS0tLmJubw7lz57TV2Ww28eDBA+RyOfz5n/85bt++3RVvsyxrm9tIq5M0pPHxcVy4cEF7JS72DjbooLHC8JWUPSEVY69zmXzfnazVfuPEK85AIIB0Oo3R0VFEo1FtSbLrjoyDAZtVI1w0lUqlK/MbCAS6qClsTABsJoGazaZWAtFoVLt2lmUd2f9/1OCmYlmWJjdLpUk5mFaDdLs8Hg+SySRGR0e1tZhMJjE2NoZ4PK6z3qx7jsfjGBkZ6VKc1WpVL1xgy0XsdDr6s0zGRKfT0WEWWrF8v4u9Q643JyYFvw9O79vpnPI787g4aL9w4hXn7OwsvvCFLyCdTmN6ehqxWKzrBlNQZgXL+Pi4Tgaw3RVrYckrq9frqFQq2gryer14/vnnEQqFdPb29u3bmJ+ffyLL9QKBgG4qTM4ey+JM0jsATXJnQwfyPPP5PP7iL/4CP/rRj7QiYz9NAPjZz34GALqihCRqyQeNRqNQSuHWrVsANss82XOTnECZ4ZeKEwCmp6ehlML8/Pwgb+GJgjRSTMUpm1mbRQxOPGm54cpzuzHOPcCprI5IpVJ49tlnkUqlkEwmdRmXfJ9021iVQAum3W53ddxptVraisrlcl3trAKBAKanpzE2NoaRkRHE43HdfbxUKg3mZgwR/H4/4vE4YrGY5lvK2CKArgXE+0qCO4+r1Wq4cuUK3nnnHd2lncqx0WjgwYMHmvNnvlcpBcuycOHCBcRiMe1mU/kyJiorjIDtHNxEIqE/18X+sZNyo1KU3wn5mvmceU43xrlH7MTjarfbqNfretHSHZBVJBJmpYn8DMZixsbGEIlEMDo6isnJyS6+2dzcnO6os7S0pKlITyLi8TguX76MmZkZTXyXnFaZTZf8TSo+v9+PdDoNv9+PZDKpY550/YPBoLb2WQEGQCd1uJhYIx8KhTA1NYVgMIjZ2VmEQiEd3yQ1ybIsrSwZOwOAkZERXQboYn+QPF3ZXFq+9jhCu6zykgm8Qa+xxypOpdSfAfgXAFZt237u0XMjAP4XgDMAFgD8lm3bG4d3mftHq9XS7hsTPHT3JD+MP+T/kbhuVpPYto3x8XH9t7RsyBX1+Xz44IMPcO/ePWSz2aGsUR+EXFOpFD75yU9ienpa079I5ZE0FFoMMhzS6XRgWRYmJyd1fNGyLIRCIa1ASV2Kx+Na4XJRUnakGbGuORQKYW5uDmfPnkUkEtGKstPpaI6oTFaRopROpzUrY9gxzGuWa0kqO26c0qp0KoN1AgsdBr3GdmNxfhPAfwPw5+K5rwH4gW3bX1ebs5m/BuCP+395B4cZV3HikEkuIbC9XNMUoulems8D0PQlLsQhxDdxyHJlzT9/qKToFvP+sQ6dFSOURzwexwsvvKA3PnZYYixUVgnJihT+Lbl9gUBAW7ztdhszMzNadiZNhj+dTkezL7LZrONAsSHFNzGka5ZrynzOdNN7HWu+B4D2FrhJDmK9PVZx2rb9f5VSZ4ynfxPAP3v0+7cA/B8MseK0LAuWZelFxAXH+CW7trTbbZ3skdVFcjeTC4xde2QygYIOh8MYGRl57Fyco8Ig5FqpVHD37l1UKhXdni2ZTCIWi2kqF91uAMhms12JuDNnzuDFF190pJjIzWwnLp9Je2JzkVqtprsjmYR5HttqtXQF2Ntvv60b6A47hnXNOslHcmhNefL3nc7DsApj1YNiPew3xjlh2/YSANi2vaSUGu91oDrCKZdUklwYhNzFpJXC0aLNZrMrliUFSvqK07mAbheEvDLLsjSXc8jRV7k2m01sbGzA4/FgaWkJjUZDN1axLAvhcFgnkEhUlx2LfD4fEomE7qQjEzcAtlGZnLwKGUoB0BUuYFs7yr5cLuvNlJ2zaGUuLy9jbW3tuFicTtiVbA9jvT4ucbOTZfm495iVQ4PKrB96csgewJRL060GtpQW21TReiTNpd1u65stLU8qTo4hZTyU55SZX2aKaTXZ9mYfTypPv9+PRCKB06dPo1Kp4ObNm4fx7x8JdiPX9fV1vPbaa7AsC6+99pqu6Q+Hwzq5lkwm8eKLLyKVSmF8fBzRaFTf80qlgqWlJe3GM7lnJvR6hUucjqH8gU15NhoNrKysoFgs4u///u9x/fr1bW4/Qw6SwnRScRjrlWESzpWiousV/trh2roqzoAtQ8Xj8eix0GwEIntV9Bv7VZwrSqmpRzvXFIDVfl7UfiGFIUnXvJmmS71TzJPJB7O7uNk4Qlo9tr05xqFarSIYDOpuLizzPAboq1zr9fq2aqlQKIRAIIBYLKbrv8fHx1Gv1/WYYJl5rVQqADbdfqk4d2NZyMUoFxq7YJFaxOqwd955Bz//+c8P8i8PM45kzcqQlowp89FJee4FMpRmhlsOs53jflfz/wbw7wB8/dHj9/p2RfuEdPEA6Nrm8+fP645FTD6YJj3joB6PR3fN8fv9+sbLfoF0J6l4WT4WDodRq9Vw//59ZDIZpFIpxONxXds8qIqGA+LQ5SqnVjYaDRQKBUSjUaTTaYyMjGBqaqorUcB7zySbrCjarfsn3ye5uNwYl5aWsLq6qpX0CcWRrFmuGVruXFM7Zc13Sgjx0fxd9mqVxROHhd3Qkf4nNoPKY0qpBwD+CzZv/reVUv8ewD0A/+rQrnCXMLNxExMTePrppzE7O6stPi5apz6Q0g1nfE1WBMnjpZXJ3TQYDKLVamFlZQULCwuYnp7Wk/mc3MujxlHJlRZfvV5HsVhEPp/X3NiXX355mzKUcWoZxwS2d8ByUqgm5UVaJKSxZDIZLC0tnRjFOWxrlhvlQeZ4OWXd5Wvka5sNXA4Lu8mqf7XHS/+8z9eyL0jqCLC1UKrVKnK5nA74O5V0URHKprossZNuurnDMX5KJdBut3UvyHQ6DdverHV/+PChtmaGraXcUcnVKR4NoIvlYJbT9VoEfH0vloWZfWefTjabkMcNmlTdLwzjmpXWfz/PZ04RGBSOReCtF2hB0B2QKBQKePjwoe6fSAsD6O7IQzoKKTFSEdbr9S43z+QEUtFy4XU6HZw/fx5zc3P46U9/infeeQe3b9/GwsLC0CnOo4C0+KgoJTiehIk703LshyIzrVYAeuaNuVE6KXgXe4e8f2YLv4NCrulBYugUp7lz7OcmK6WQTCZx9uxZjI2NaQVJ15sVK9JKpZXjxA10uj6Te8ZdLxwO61ZzdB3cUcObcJIlrXPZ2ENamJSLPN5MLuzGOjTfJ5+X1UrEcbY4hw1O1mA/7625Jk8EHWk3oBXA3+WjtCbNm+1kaQKbu8+v/dqv4bd/+7d1eR4XJ11z7nxMFsi5MwTJ7aaryM+UhHfG4djOLBgMolqtbnMBn3SYPQBarRbW19d1o2dOO2RMU95rwkl5mjCflxVJMjlIShkLH/jex4UJXOwe0tMgnKhFJkwObq9zS+/RfO6wMBSKU8JJce60OGSgn3+zxvn06dM6aMwF62TByJsu3fidBObkwimlNAWJWeBBx16GHaY8GWOkxWm66VJee72XpqstNzl5LsnbdNF/7Gcd7FYW/fBQ94MjUZw7Ka5eis0JqVQK6XRau+WRSASTk5OIRqP42Mc+puOPlUqlSxHKDDk/mzFImS3n77R+JJ0F6CbfMivP2UKTk5P46Ec/Co/Hg2vXrj3xky4JuYikxc9sqKQKmQwG8/29zitj33xOUstMxenkVbjoDyTzRFIGTbfaTMztpGydWBMmQ0Y2fTkMHJnFaVqRvX7fCeFwGOl0Wk+iTCaTuHjxoq6HltVAZvZcKk02igC6g8xcZJLHCWy5m9LV4GLlAo3H45idncX9+/ddq9OAkxtNV9qMNe8Uc94t30/KW3omThupi/6Da8hMDEnZSsVHPM5Fl+cwP29YCfAHRq8vKonobMzBmmY5r4Td2ScnJzEzM4NoNIrp6WkEg0EkEgkd16Qy5DwhuoIkwcsFJhvU2ratR5RWq1VsbGx0Kd5EIqGb88pFJ2MtsVgM09PTevibiy04xap7Ka9eitO0UOQjfzdjaOZnuJnzwcNMwAIHd6+dEriHbawcieLsdaNYb+r3+3Xvw1gshvHxcQQCASQSCYTDYXzkIx/B3NwcRkZGdNacZrkM+rPmPBwO67kz7N8nY6Occsldir0gfT6fHgcMAMFgEJZlYWxsDMlkUi840mikFcu+nKOjo67i3AUoQ7PN304WSK/kglPWXb5HhgCA7iSfPMZVpv2DU7jFycrk63s5r8SgusEfieJkV5xkMolUKqW7F3m9XkQiEd2cg8PQeEwwGEQgENAllLL+W1p/tDaZMWXCgRYsaS+MS8pFwt6QvKZAIKCb11Kp87yy8S6tW7qBLMN8kge17QWUDwfhSQvTXAi7Sd6Zx0qY1uogZ9U8idirItutbJ0+43Hx0X5h4IrT6/VifHwco6Oj+PjHP45PfepTCIVCXQpTzgWSiQJadTLAz0azMvmilNLKsVgsYm1tDZFIBJcvX0YkEsHDhw9RLpe7Eg/sGE6lLG88FS+7u7Tbbayvr+vPoKKlxczOSrSa3UX5eCilEIlEkEwmtQXP56VLt5M1+bjz8z3tdltvulScrvI8POxHmfWSreneS/mZP06bZr8wcMVJ5TQ+Pq5jlIFAQE8aZKdvmSigsuLzkpMnf6erzpvu9XrRarWQz+d1N28qZTY2lvwyZnjZao5UGZn88Xg8ulcjM+hOSSf5/7roxk6hGpay7sZ62M29dYpnmtcwCAvlSUc/7+9OynBQHsTAFWcwGMSnP/1p/Mqv/IpWnPxnga3ZysyIA1vWhUlnYBxTuuh+v1+3lLMsC6urq7h27Rr8fj9arRZSqRSeeuopzM7Ool6vd8UmG40G7t69i2azqcc9BINBbQ1TucvBYGYSgs9Xq1UopTSh28UmZBLN6TUzvkkqmJSxk9Uhz8Hn5EbGxSQz+DzWtTgPF1JmZkLVvO9OspQwvQ7zczhXCoD2Rg8DA1WcVGzT09N46qmnEIlEEA6HAWyvxnGqZaZl1yuwTwExNhkKhaCUQi6Xg8/nw9LSEmq1Gi5cuKBdaJlpbzabyOfzqNVq2NjYQKFQ0HPY/X6/VspceJKqJCE5qXTzXTwelLuEaSma2IurLj/HfM2J2+nicPC4pi2PgzSinM5BPXGY626gijMcDuPy5cuYnZ1FKpVCq9VCLpfrqvyhhcHqG4I3RCrHWq2mZ8hUq1UAW3QmPj8xMYEvf/nLyOVyuH79OhYWFrC+vq5LIyW53ePx6Bnb8XgcY2NjCAQCCIfDjg105ZRMqcjl+cgG4Bx2F85g2IWbkrTgTQvDKRzCY83XnGbamMcEAgEEg0HHTdDFwSF5nNK7dEr47QdmYpjdyw4Tu+nHOYfNaXmTADoAvmHb9v+n9jFuNBgM4tKlS5icnEQsFkMul0OpVNIWIhWTzLKL69A3n7SgYrGobxjderaQY3xydHQUZ8+excLCAn784x/j7t27uHXrluZjMmPPTvFnzpzRSnN8fLzL4mGlEIViXiOwPWjNDvC0fIcF/ZRrv8AQjXTHpDtNmIpQyqjXa3xOKlgAXWGdk+AZDKNcpUUvf5ws/70qT6dzUH8ctcXZAvCfbNt+UykVA3BVKfUPAH4Hexw3ysx4pVLRTRXkHGsmcABoHiWwxc3i+ItarYZGo4FSqYRisdiVUZcZdrkDeb1efPSjH8Xp06d1eV88HsfIyIiOi/j9foyOjiIUCiEajW6LqfJHNoMwq4r4O6+Zc3aGkMvZN7n2C4+LR/VabNIifdxiMa1OsjTIinA69phh6ORqJvlMOe3nPpvfFZP/e9jYTSPjJQCcjldUSt0AMIN9jBvl0Kt8Pq+nHwYCAU0ip5VIAjrdbklJ8ng8umdmoVDQrj7jj7RYmFwiZSgQCODLX/4yAGB5eRn5fF5bnH6/H5FIRFuqTBTxWmTSQnZq6uUGygUeCoWQTqdRq9Vw/fr1vcjmUNFPufYT+w3myzaBEr0sT8nIoGsnN8rjan0Om1xpbTqNqzFDJruF0/sOezibiT0FApRSZwC8AOAK9jFuNBgMdrW2l7u+rP3mF1jGu5iMYWaUilG6cnJnk8J6dB1aEZPQHolENJGelqOsXWfMjddrVraYCSwZTpC7K7Pxw4qDyrXP17Ite77X9+8GcvFJz0G+ftwxbHLtJZt+bFRSnoMYE7xrxamUigL4DoA/tG27sIcvqB43GolE7Fwup5tukA5El1bWo8vGts1mU7vc8oaQ28lYI7PfjFvJcRlKKVQqFSilEI1GEY/Hu6zLarWKTqejQwgUNLsryR2Nj5JSQaslGo3qTjA8B6lPw4h+yFX1aYzs48rleikz83293m9anLRQJWPipGCY5GqGQszQl9zA9gq+VxoxbO145IpTKeXHphD+0rbt7z56es/jRhnDpFIkFUi6WWYyRpYzSouUpr7M1EmLU+468nxU0n6/fxuRnb07ZTkmFbepOOUAKipsJpmk1T/moCsAACAASURBVAtgWyXSsKBfct0rdmvN7fae7aQ0d7JeZQ/Ww04mDBJHJdcdrsfx3u7XVXcCz+VkYB0GdpNVVwD+B4Abtm3/V/HSnseNNptN3L9/Hzdu3IBlWZiamsKpU6f0aM9Op4N8Pt8VYzRnJUvrUipMSZSmwpUWBJ/vdDool8va/TetSGmFAtDxVr6fljIVeq1W09xOOcqWHFXGYodt5lA/5XoI17ajIuSjuSDNJIT5Hh5jWj7yueOOYZSr7Fcr48umXHYLp8QS17LP50M6nYZlWXjw4EH//gkDu7E4PwXg3wK4ppR669Fz/xn7GDfKGu+lpSWkUikkEglEIhE0Gg1dwcMORvzx+/268YOT8pQKFIBWjiZ9RSaepFlvuguyu1Kn09FuhhSQDBXIruV8bDab2srkpjBsihN9lGu/8bj45k5WyuMWoLnoDuImDimGTq5OlVly0+Lfe1WeMhHLNc6QC0uhDwu7yar/BECv/2jP40bb7Tbm5+d1RjyfzyMajWJyclJbagB0/BPYUoayWkcqUNOKALZnUYHtM4J2slhk/FKWgdqP6CuMoVJB8lrC4bCuqV9fX8fi4iLef/99/b8MC/ot136AMgawI4HZtDadLJfHuYByw9xvImoYMWxyVY+SstFoVBtAfN7EbjYxU9FKOXON0hA7zK7+A69Vb7VauH79Om7cuIH79+/j3r17OHPmDD73uc8hkUjo5Eq5XNZJIlqh5lwgJ5fdzGibmVPz0XzOSWhKbXZPYpKI72MPTwBaocbjcViWheXlZWQyGdy9exe//OUvUalUBnSHjy/4xZcVYyacrMTdLBAni8ZUnCeExzlUUGpz8mssFutqsWhucDsZPPI9prLk85Rls9lEuVzWyd7DwpE1MrZtG6VSCSsrK1BK4a233kIsFkM6nUY4HNYxQrZpA9D1BeejqTjN2JjTWAxJJyJkQxEn5cmYptO0RDYFsW0bKysrAIDFxUWsrq7i7t27XbQpF7urENnJcpSeAY8zeZw7fYapIHeyflwcHKan1yu2+ThvoVcyScapzdzFYeFI2srxH1pbW8PGxgZu3LiBn/3sZwiFQrh48SLGxsbw6quv4sUXX9S14sxYS26mjJ04BfyBTZe/VCp1uWaFQkHXsgObSpEUqV5JAipgOYKDC3djYwM3b95EuVzG2toayuUy7t+/j8XFRT1b3cXu0cuy4GvyGLnwepHgCacFa27ALvoP2cWM7ReB7go7YEs3yF4FUiZmq0m5psmU4Xo7bGPlSMcDk85Tr9dRLpdhWRYSiQSazSZWVlawsrKi27ox/inda2ltMCZm3rBqtYpCodDVsYjzu4Gt5hL1en2bIPm6PI7ugCS+53I5rK6uolwuY2VlBZVKBaurq1hfXz/cG3hM0esLLYsMeh27U1LBVJrSQtmtheuivyBDhUwUYLvnyOd6vb8XpMEkDSrSDE+U4tzpn2m1Wrh79y4WFxexvLyMv/3bv+1q7JFOp3W7OI7AqNfrsCxLE9oLhUJX9rxUKmF9fb1L0clkEyF5pL2uu9eCZlyFCpiZdBe7h23bKJfLyOVyiMfjCAaDALbPPDeTdtJKcTqOMpOVYfIz5WbpKs/+o9PpYG1tDaVSCUopTExM6N4QpPiZIRYZswS2x0EJ6gXZ4azRaGBlZUXnRQ4LR2pxmqCiA4CNje7GLYFAAFNTUwgGgzpDV6/XUa1WEQgEkEqlAADZbFa74Z1OB8ViEZlMxo1ZDTm4iBqNRtcmZ9LFzL8fd5x83cm66ZUYctEf2PYmxZDjUMh3Zl9d875LN13GKXutX7roMm9RqVT0TLDDwlApzp3QarWwvr4On8+HjY0NPWKBVT6ZTAbApmsuEziMXboYbpC2woF8LDrgIuKoE3m8ZFc4geEVYMvirFQqqFarmlLWbreRSqU091ae3/3e9Ac0YhYXF9FutxGJRDA+Pq45l7KvLbAVqzY3QqeMO2eHZTIZbGxsIJfL6bLpE+Wq7xeswHFxcsG+rGxiLelnpKmZGXWl1I6cT3kOADrr6vP5tOKMx+Oo1+vbaFCu8uwPGBZbXV3FxsYGUqkULMvSNCXKkPJ1UpxAN+OF+REmeu/du4f79+9rr+WwcWwUp4uTDdu2UalUsLGxoUMwZDsopfQQP2B7Rn0n+orsugVAc/xYkVYul3H37l1ks1kdJpLvd9E/0AOoVqtYWVnRHgYZM+zBywYdphsvKYHsKVEsFvXIGzNeephwFaeLoUCn00E2m8X9+/d1pykWHdi27dhomJaJzNiakJ2sAOhsq9/vRygUQrVaxfz8PIrFYleHfldp9h+0PPP5PIrFYlfMmZxtjtX2+Xxd40y4sZZKJTQaDeTzeb2x8ryDUpqAqzhdHAF6ucB0vwh2pgK2V/7Q4qTiNLt/8zPkEEDzM3w+X1frwkEuvCcNUuZSLgRryykPYDu9TDYnJyVwkFamhBrkzqqUWgNQBpAZ2If2D2M4+HWftm073Y+LGSa4cnXlOoQ4VLkOVHECgFLqDdu2Xxzoh/YBx/W6B4Xjen+O63UPCsf1/hz2dbvkNRcuXLjYI1zF6cKFCxd7xFEozm8cwWf2A8f1ugeF43p/jut1DwrH9f4c6nUPPMbpwoULF8cdrqvuwoULF3uEqzhduHDhYo8YqOJUSn1eKfW+UmpeKfW1QX72bqGUmlNK/VApdUMp9a5S6g8ePT+ilPoHpdQHjx5TR32twwJXricTrlx3+NxBxTiVUl4ANwF8FsADAK8D+Kpt29cHcgG7hNqcOT1l2/abSqkYgKsA/iWA3wGwbtv21x99iVK2bf/xEV7qUMCV68mEK9edMUiL8xMA5m3bvm3bdgPAXwH4zQF+/q5g2/aSbdtvPvq9COAGgBlsXuu3Hh32LWwKx4Ur15MKV6474ECKc4+m/AyA++LvB4+eG1oopc4AeAHAFQATtm0vAZvCAjB+dFd2uHDlenKxB9m6ct0B+1acj0z5/w7gCwCeAfBVpdQzO73F4bmh5UIppaIAvgPgD23bfmIagbpyPbnYo2xdue70efuNcSqlfgXA/2Pb9q8/+vtPAMC27f93h+N/us/r7AJbjFmWhUAgAJ/Ph3A4DI/Ho9uIsdO37Ono9/vh8XgQDof1ewOBQNd8mlKphGaziVKpdBhzgzLD3gziKOXab8ju8IfcEXzo5QrsTbaHIVfOifJ6vbq/ajQahWVZui0cOx7JVoDs9G9Zlm4+Xa/X0Ww2df/WQ5JtT7kepK2ckyn/knmQUur3Afz+fj+EX34qPq/Xi5GREUQiEczMzOD06dNIp9N4/vnnYVmWnjfyzjvv4P3330c+n8fDhw/h8/kwMTGBSCSC559/HjMzM5idncWZM2dg27aetPlP//RPWF5exptvvonr16939frrwxiOuwd584AwELk6gZugbB/HBcRZRDvdf76PCywQCCCRSADYnERaq9XQaDR0q7o+4jjIFdiFbPshV9mZn8qSBks4HMbIyAieeuoppFIpfPrTn8bk5KRWhIVCAWtra7qFHACtZKempjA7O4tCoYDbt29jbW0N3/ve93D37t2ukcBcr6YC3gd6yvUginNXprxt29/Ao/InpdSe/4NkMomZmRkkk0lcvnxZNznlOIVYLKa7R3s8HsTjcQDAr/7qr+LVV1/t6gLebDahlEIikUA4HIbX69UCqtVq6HQ6OHfuHGZmZvDMM8+gVqshl8thcXERmUwGP/7xj7ua3Z5QDESuJgKBAL7whS/g2Wefhd/v15ZFrVZDsVjEj370Izx8+FB3cN920Y+a4VqWhU9+8pP4zGc+o8czdDodrK+vo1Kp4Ic//CFef/11bdk8YXisbA8qVzYgjsfjeP7555FMJjE1NYVUKqW77vt8PkQiEViWhfHxcYRCIYyNjSEUCnWNiKbC44ZIy9Pv96PVamF8fBzJZBKVSgW1Wk1/V5aXl5HL5XDt2jUUi0WtVPuJgyjOBwDmxN+zABYPdjnbEQqFMDU1hampKbz88ssYGxvTjWs5T50D75VSuhX/5OQkRkdHdUfpZrOJTCbT1TG6XC6jWCx2NbJNp9Pwer2Ix+OIRqNYXl7G+++/j7t37+Lq1atPguIciFxN+Hw+PPvss/jMZz6DYDCIUCikQyaZTAY3b95EPp9Ho9HoqTipKC9duoQvfelLXbOLOBpjYWEBv/jFLwDgSVSchy5bzo2Kx+N4+umnMTk5iUuXLmFychJer1ev13q9DmAr7JZIJDAyMqKVq/Q6qEiLxSIKhQKUUhgZGUGn08GpU6fg8XhQLpdRr9exurqK+fl5LC0tYWFhQY/Y6DcOojhfB3BRKXUWwEMA/xrAv+nLVQEIBoMIBoOYmprCpUuXkEgkUKlUsLa2po+hC0BrEticcsmdiQO4AoEA2u02yuWy7hjNsaWch043nG5GPp+H1+vV50skEvj4xz+ObDaLmzdvYnV1tV//6rDhUOW6E9SjOTMca0EFGg6H8Ru/8Rv45Cc/iWKxiEqlgkKhgNXVVViWhZmZGX2sZVl4/vnnEQqFdIf4drutZ9kA0N3Dn0Acmmy5Xs+cOYOXXnoJiUQCZ86cQSQSgc/nQ71e75oRZYZLqtUqlpeXtbfBsSjAVtf+ZrOp45ntdhtKKbRaLT1KWCmFSCSCubk5JJNJAJshmjt37iCTySCXy+lpuAfFvhWnbdstpdR/BPB3ALwA/sy27Xf7clUAIpEIEokE5ubm8OEPfxg+n0/vOH6/v2sIPeMYsiV/vV7XUzHNY/hTr9dRq9X0++jW00Ws1WoIhUKIx+MYGRnBq6++ilKphFKpdGIV52HLdSdQnpZlIRKJaAtFKYWnn35ab4aNRgN3797F22+/jWg0ipdffhnJZFJvmDLGxVGxHAcMYCBTEIcRhylbrtcXXngBv/d7vwfLsrQ3x1CYdMH5yOFr1WoV9XpdD257dL1aydLDpByj0aj2OJkHYagulUqh0+ngqaeeQq1Ww09+8hPMz8/jgw8+QDab7Usi6UAzh2zb/j6A7x/4KgwopZBKpTA3N4fR0VG9CMy5MjyWi0Wi0WjoQV9UpnLHo9A4vIuLTWbYeQzdCi7oVCqFsbExbbGeNByWXB8HykPOCWq1WnqBeDweBAIB+P1+NJtN3L59G4lEAh/+8IcRDAYRDofh8/l0TEtalbRYnvS5QochW6UUxsbGcObMGUxMTOg1R6UpjRJzfck4JtenebycVMrNUa5hep3SA+XfXq8XY2NjaDQayGQy8Pl8WoEfBEM5rM3j8eDSpUt45ZVX4PV6NS1I/rPcZXw+X5fio8VYLpdRLpfRbDZRq9V0lk/edN5YYGuqoRQWh0etr6/DsixMTk4iFovhwoULAID79+/j1q1b7kTEPsC2bdRqNZRKJUQiEb34GLu2H02mpEu+srKCb3/720in07h48SJarRYmJyeRSCTQbDZRqVQAbC3IWq2Gcrn8xFqbhwmPx4OPfOQj+OxnP4t4PI5qtapzCK1WS7NipLdnJn+8Xi+CwWBXNpzuvKQxBQKBrrnrdNN5HqmAOS312WefxaVLl1CtVvH2229rr+Ug63YoFSeArt2FN13efMY0TEHIRwqBx0rr1LZtrTQlnHZFJ0vXRf/BWLO0FDudjl4cdMfIhsjn87AsC+VyWbMipNUieZwHpKW46AHKIxKJYGRkRLMhpAdHNxvozpBLRer3+7ssQRouthgN7ff74ff7u3SDCZ6P69zj8SAYDOr4ZzgchlJKe5r7xVAqzk6ng9u3b8Pj8WBubg6XL1/WmTPGH9vttk780Mrke6lYefOj0WjXTsRMmzmOVn4+z0Gyrsfj0dSGDz74ANeuXUO1WnUXY59g27ZO+DBeacaunUbBtttt5HI5ZLNZzM3NIRQKdbEuaO2Q2+m0WbrYHzweDyKRCEKhEBKJBBKJhGa40KOjccO16PNtqhyuRT6aBo8MsVFZknYIdHuI8v30UgDoxBEtz8nJSVy+fBlra2t45513dAhuPxhKxQkAhUIBDx8+RDweh9fr1T+8OXS5uMBMcjSVqdfrhd/v7zq3FFYvUIA+n08vvlqthmq1qheqi/6Bybpqtaply0XH13vNTq/X6130Fvk+bph095ysFBf7B40X/nDmubT+pSykHOSPkxvP9cnCF1qcALZ5g/K7YRLf+bmRSATpdFrPcD8IhlJx2raNXC6HZrOJaDSKe/fuIRaLIZ1O65gmg8IUAp+nq8cfGcvoFRRmPIW7mryppLEUCgW89dZbWF9fx/Ly8sDuxZMCWpxra2s4ffq03hD5GhcjPYVAIICJiQnEYjG9mGQCSCpMvv+wOH1PKpisC4VCiEQiiEaj2tqX1D4aHlImMu7pBKn4ZIiGxhPXsVNugmufjBkq3tHRUZ2foOW7Xwyl4gSAYrGIYrGIRCKBpaUltFotzQvL5/NdpVym4pRJIhlspiVDl43vdyrvkkJjpvbatWtYWlpyJGC7OBhs20a5XEYmk0GlUukKrUhWBRWnZVm69FZ6ItKd5yLl+/jdcNE/WJalGQ2RSAQej0dzLanoLMvSxHeGULjZyZgkAEePQFqohPk+qUyBrRg3ObxerxeJRAKnTp1CoVA4mRanRKFQwPvvv49KpYIXX3wR0WgU4XBYZ01lkogLhLQWKkP5OhWodBsoUBJqCRJ3GXurVqsHDiq7cAaz6nTVZehFbpKtVgvVahXpdBq//uu/DsuycPHiRSQSCV2wQO9BLjST6uTi4GDZczKZ1NU+TMYA6OoxwdcoV2bea7Wa3tQkNRBAV94CQNe64/nC4TD8fj/i8TgikYg+XobZaChRgfcjzj30inNlZQWZTAYf+tCH8JWvfAWjo6NYX1/XlBMmaPhDd4xNAwi5kGh5cDECW7ExLloGovP5PObn55HNZlEoFA6jY5ILbC6KSqWCXC6n+bcyTkZZUb7nzp3DH/3RHwGAtjLZW0C66jIOWq1WXVe9j/D5fBgbG8PU1BTi8bhWmlRMXFu0GJmsqdfrOtG6tram8waFQqHLM6Bcycvlc/QCLcvC9PQ0YrEYnnrqKUQiEZ2cAqCVOevjS6WS5gEfNNY99IqTu0epVML9+/fh8XhQKpW2kdmlay5dNTNITMj393IBKOSNjQ0tVBeHB254Ttlzgi4541bAlmVi8nn5nZCxUhf9A7ma7EA2Pz+vFRoflVIoFos6gUcOJRXn+vq6bs5RLpe7FCZlx79NIjzjlKFQCF6vF/V6HdFoFCMjI11VZ7R8eU0nWnHKwH6n08HKygr+9E//FKlUCp/61Kdw7tw5fSx3JRnPZBxF7kDc+WSAmZl3J26Yx+NBLpfD22+/ra1bF4cDWpz5fB6VSkVbJ0D3JsfNq9lsolgs6tdl9Vij0UCj0dALmN8JN6PeX9Trddy4cQPz8/O4c+cOvv/972N2dhbPPvssIpEIJiYmoJTCa6+9hlu3biGXy2F9fV2vU2DLW+Dak4kjKj6618xLMNnXbrdRKBTQbDa1JfnKK6/gd3/3d3WTHsq/0+kgEonojkq0hveLoVacEvV6Hffv38fGxgY+/OEP60Uiqwicyriczvu4YDSwZb1wd3Rd9MMFlSJDJU7JAD7KTQ/AtqytfF2GcVz0H2Qx0HL0er0YHx9HrVbTNeeLi4tYWFhAqVRCPp/vej/lQg+ClqIkzNPQAaDjpAC6ekqUSiXYto1MJoNyuQyfz6d7WtDVp8XbjzzF0CpOxqkklahUKqHdbiObzWJtbQ2BQADhcFhzx7hgZGxTEmJplTCBRFI0IV101qFTIC4OF0wOlUoleDwejIyMaDlJ1oOEWY3ixPXs5ba7ODhCoRBeeuklTE5OasXXbrexurqKtbU1LC8vo9PpoFAoIBgM4mMf+5guZgG2aGZSxub6BdBljQLoUq7sq1sqlbRX+O1vf1uvZSaimHxie0l6K/vF0CpOoDsmxQA/AK3QvF4vYrEYgG53zaw4abfb2jJluVWr1dI7kqxS4LlY496Hru8udgnZVyASiXS1IpPxLYKWJiHjYIS0UGX828XBQUbDhQsX9H1eWVnBzZs30Wq1kM1mddKOx37xi1/UcpNZdZbNst+qdOdND9Ln8yEUCiEQCGBmZgbhcBjZbBb5fB6vv/46/vEf/1E3N2bfinq93tfm1UOtOHcCTXjOKyFo1ssEA5MIkuZgkuHNxcbuSiTRujhccGNkeISQpGnzOTPMIkv8TJ4vCfCu4uwfWq0WlpeX4fV6kUwmEY/HEYvFcP78efj9foyOjsLr9WJhYQEbGxsoFov4wQ9+oEsgOTeMSpReAeVvVn/RwKGFqZTC0tKSTkCVSiUUCgW88MILOuzD1oKNRgOJRAKpVArLy8u4cuWKbgK0n+/EYxWnUurPAPwLAKu2bT/36LkRAP8LwBkACwB+y7btjT1/+j7BG8nefTKrymybtDQlQVqSqJ1iYHys1+solUoHqmcdZgyjXGu1mo5DmZw9ytBJYZoLzKxSIc3sSaEjDUq2rVYLDx48QK1Ww/nz5xGLxZBIJDQ96bnnnoNlWbhy5Qru3LmDxcVF/PVf/7XuhBQIBPToDGa7ge6QGdcvY5vSCm02m1hcXNQWZaPRwNzcHF566SWdFGq321pxXrx4Ec888wzefPNN3L5929GA2i12Y3F+E8B/A/Dn4rmvAfiBbdtfV5uzmb8G4I/3/On7hCyfc1pETjXN3NVkth7obg4gE0YMJpuu+k5lYscM38SQyRXAto1MPr+b9/Z6vleTkBOKb2IAsvX5fHroIecKsabc5/PpWGU0GsXExASCwSBSqZQu1fT5fEgkEvp3bo6ScgSgKyZqVgfF4/GuDTeVSiEajQLYNH48Ho/2HtfX1/Hw4UOsrq5q/bHftfxYxWnb9v9Vm4PeJX4TwD979Pu3APwfHOICMzmb5XIZGxsbSCQSXRVALLGSzVMBbOvZKbN2NNUbjUYXcbdWq2FjY0OTsfm8jKMeZwyDXHuBCYLdVPlIRSs3QzMmSovzECZcDh0GJdtwOIxXX30VL7zwQte4mmKxCI/Hg2w2q7Ps09PTSKVSGB0ddfQaZHd4U+4mtxpAF1VJKaWn2xYKBaysrKDRaOiClVwuh+XlZaytreG9997D/fv3kc1mu9b2XrHfGOeEbdtLj/6pJaXUeK8DVZ/HyNJENxs2OMW9TIqStDZl0sGkMfF500J5AniARyZXE72szsfJoNem1quF4BOEXcl2r3KVRHeuLZZcyi5XlFuv7kZSPk5rUSpOeoSS8yldeh7LiQAMBbAakBax5PjuFYeeHLL7PEbWtm2USiWsra1hamqqq4qg3W7rm1QsFvWgNRJsg8GgtiapECU9iaA16tJXeqPfcjXOvWvupeTlOh0vKWqHMSb2pGEvcm00Grh58yaUUpienkY6nUYwGNRjZRYWFrR3WKlUEIvFkEwmtTUoS2rNmCafl0PbgC0laio82UHJ7/fr+UfhcBhjY2NYXl7G5OQkZmZm8NZbb+HevXtYXl7GysrKvgpb9qs4V5RSU492rikAA51c1mg0NI0B2LrpdLVlTBPYslR4c03Xn787ZdmfMMV5pHLdDUxqCn9/nJxcKlL/ZUuOZiaT0eN92Tgc2Fyn7HiVy+UQiUSQy+VQr9c1VUmuXan4ZGJPfh4z70za0rihJRmJRJBMJhEOh5FKpZBMJpHP5+Hz+TA7O4uzZ88il8shHo8fqEvSfhXn/wbw7wB8/dHj9/Z5nj2DZj25mKZlIv+Ws0pkmSUbPsjem7LVHN8r/35CcGRyNdHLJTez5sAWzcwpUcjvAntGHrQP4zFG32VLGtL4+LhuYsy6c9av5/N5BAIBTE9PI5lMYnR0VE+1JAvGiQlB9JKp9BhpgbZaLRSLRSwuLqLRaGBxcRHVahXZbBbZbBatVgu5XA7Xr1/H8vKybha0H+yGjvQ/sRlUHlNKPQDwX7B587+tlPr3AO4B+Ff7+vR9QmZId1Kc0vSXs2cYIyWPjApWJo0kD/AkYhjlKq7NMYHgdJz5KEtw5QJjlZk5DeAkYlCy9Xi2xvFalqW5k5VKBcViESsrKygUCjh37hzS6TTGxsZ0/bqZcCWcqIG95CyPZ5OQhYUFrK6uotVq6cf19XXk83mUSiUsLS3h1q1b2go+NMVp2/ZXe7z0z/f1iX0Ab5as7pFD6+Vo2F4ZWq/XqysXOD1P8sROuls3jHJ9HMx4mJOLbvI3+XowGEQkEnkiFOegZKuUQigUQiwW064ymxqXSiXd+pHrT4ZLzMmUpvEjH+XnAeiKg/I8sqUd1zHdfs6hYv16LBbTPX33S1E7Fn6LU3aVcY5SqdTVuLhWq3U1MeZrJNBSUH6/Xx/v9/v1DuY0k9vF0cApjmkqT2B7FlYqTi6KWCyGTqeDUCg0wP/gZMPj8SCRSOgKIdaOJ5NJXXVXKpW6asW5dlmRx3XH2KX08kyKmaxVZ48JGePkDymGVOQc1hgIBBAMBrG6uopEIqEbhOwnYXgsFKcJBpPl+AtakDKwTLoBsHWzZRKJRf8+n0/vVlSirEoyk0lPACVpaGG6bPL5XpYnwREPJzX0chSQiR1zkwPQ5cH18hB2WlM7FTT0ug7GO6VFKzslsUGI3+/XYbr94FgoTqm4AGjzmwF/Uk18Ph8ikUiXgGiVAtALh5UKFKy0UKhUi8Uims0mCoXCtiSEi8Gjl9KUr8sFJS0ZpRTi8biOc7roHySDRXKf6c2xaQuTr7QiybeUHa7M2LaUtdlazgzX0IBiF7VwOKxlz7nqNLYCgQCSySRqtRoymcy+/u9joTglpJIzg8oAtmXCO52O7gvI1ldScZpznnku7khPcBZ24KBs5Syo3bzHhOzZSDC26c5VPxyYhSSyUg/Y6mRlhlTMR6d17bTO5TlMypJk3ADdDBkqUFqg+/Ugj4VWMAPHpDysr69jaWlJt+DvdDra/GaySHZioRCLxWJXww9m1zudTlcjiEgkoueWuDhcR+Al3gAAIABJREFUKKWQSCQQCoUQj8f1VESz3M4pxmk+mpVhtDrM/qsuDg7ZK9PkRFerVU0uZwhMylNiN8khuuFyCq2Me0qLk20JZWJKfk8OGro5FopTQmbTy+UycrkcGo2GHtJFl4A3LBgMIhaLdVkhcpJis9nUQWTbtvX5EonEk877Gyio3GKx2K6/0E4xMz5PdDodbWEArsV5GHAyLGS1FrDdE5Sy26l4QSpZacn2ShTKrvDmrCKptGlQnWiLU4LdwaenpxEMBrsK+WWHI2bHLcvSs7eZOOJNZYzT7/frage+5vV6EYlEEA6HEQ6H4fF4trU7c9E/KKX0oC2/36/pLOZMGnm8k8VpyofvZXhmZGQEMzMzeqKmK8/+wIw5kuTORhqSI22+D9heuOLE4TQ/Q2baCX62GRIw8yROYYG94FgqznQ6jTNnzqBWq+lZJ6wM4NA2Duzy+/160YTDYSiluhYkJyay3RWwlYmLRqP6h++Tyhl44koyDw101cfHx+H3+3Ut805UEdnQwZQDQzP8YcfwiYkJzM3NIZPJoFAouLSzPoHuOrClvCqVCsrlss4Z7KSkTBfeaZN0WndOj2and/leaa0ehGFxbBSnUkq71JFIBNFotGtn4kxl/s1W/KQa+Xw+xGIxPUZU3ljWuPLmko7ExNLExATy+Tyq1arezXq5iS72B4/Hg2g0itHRUc21dEoE7OaeOy1QeguxWAynTp2CbdtYWFjoy7W72A7btru4m2YVnlR0ptXpJONenoSTZSkz7GZ8kzD7VuwVQ6045Q31er067jgxMYHp6WkUCgVsbGyg1WohFotBKaWHN2WzWRQKBc3HtCwLY2Nj8Pv9OsbJagen0kq69MlkEi+++CIymYy2bnlNLvoHj8eD6elpPPPMM5iYmOh63gzsO2VazVJbqWTb7TZyuRy8Xi+mp6fxmc98BleuXMEvf/lLt1vSAdGraTiTuJVKRceZ+bypHM0MvFPiyIyJmpajdPO5puv1OiqVyrb4Kg2jJyLG6fV6EQqFEI1Gu7ogkd/Fm0Prkg08SHaVLjtdbr/fr3l9UmhKKd00gJZQtVrV9CS33Vz/wU2PfEsnTmav9/V6Xr7G0EwoFMLIyEiXh+Gi/zCTMU7xxMdRjeRGuJMl6hS3VEppi9csv+WxB8FQK075jwaDQVy8eBGpVAq2bWNxcbFrp6rVavB4PLrhQL1ex8bGBqLRKEKhkN6VOp0OSqUSSqWSbrPPgU78HM5iprKMxWJoNpsYGxuDUgq5XG5fPfxc9IbH48H4+DjOnz+PZDKp5SrjmOaXn3XJTouAlghZEbRQUqkU4vE43n//fbeKaECQFXm9FJ2TO02YcUx5rKyBpyFF3vbGxgaCwSDGx8c13VBayAdx1Y/NN8fn8yGZTCKVSgFA1yA1BoS5iBifJI9LglwwjguVWXg5XVEmJkiGZ5bdpSj1H7Q4OYPGaYGZx/cK8Eu3Xrpptr3ZnTyRSOhEoYv+YCcPjKEwpxi1k1x3Oq/p4puWqPyuMFQge1T0y1M8NhrA5/NhbGwM6XQaxWJR04aUUro5AOMWkvogg8WEvOG0QnmOcDiMUCjUNaaUSpbW6cbGwAY/PjEgHYkbo8nVcyJN74ZOwmMYYwsGgzrByNplN7O+P0gjhNU5wPYsOC3OXvFN85xOLrV5Thm/5t8yudvpdFAsFhEKhTS90Az/HGTjfKzFqZSaU0r9UCl1Qyn1rlLqDx49P6KU+gel1AePHlP7vopdwOfz6WFPfr9fxxllSZfsIi2uXytPp+yd7Cpt27Zuk0WXHdga6BYOhxGNRk9Ea7Jhkau4Hl015GRxAti2oKQV42RJyE2TsW7KMBQK6Q32JFmeg5Yru4+ZOQKp8Mxet4RpwOzwP3Wdq5cRJI9nBRGrBPez6e6E3bjqLQD/ybbtDwH4JID/oJR6BlvjRi8C+MGjv/sONu4IBoO6YojJIO54shMKsJWFl5QDmYmTQWspUOnyc/az3+/X72OZ1gmpPjlSuRLqEc2Mw7R4f80BXeZ7gO7Yl6lk5YZp2zay2SyWlpZQLBZ12V4ymUQikTgp8iQGIlfT+pOusFkr7mS8OJ3P7LLkdMxOfSqAbkuS1YVM8jKWyuMOlY5kb07G43S8olLqBoAZDGiUbCAQwMjICGKxGCqViiZGMztOd9pUkFyQclHIGURScUqqBEn0fr8f8Xgc1WpVcwBDoZD+7OOOo5YrQSZEOBxGJBLRNcYyXr2Tu25uhMB2C6XZbOL+/ft69kwikUAwGMTc3ByCwaBeXCcBg5CrXDfm8yx5pAyBLfqPbDEnrlcfw7970ZKkcjbXstw8uVnmcjkA0K669E4Oqjj3lBxSSp0B8AKAKzDGjQLoOW5UKfWGUuqNfV3go3I5NmfolREzzXenHc5MFDy6vq4djMrYtu2uLwe/LLKv30lx8Y5CruI8WnHKhtKm62e+x+mHr0nI74uUGSvJTnKPzsOWq2n5S4tTJmWdZLhT0qeXG/04l57vpTw51FFm4QeeHFJKRQF8B8Af2rZd2K3SsA84Rpa98+Lx+LaefDJ2QqUGdC862T2FtCTZ8EO2nOp0Osjn86hUKkgkEojFYl3zmzkvmtYRBXOccVRyJSzLwvT0NMbHxxGJRHhuHTahTLkouajMptN8n3GN+j3kbrJ6LBKJYGpqCp1OB/Pz8/u9/KHFYcvVKa7o9XrRaDRQLBZRKBS6LEynuGSv88pzyud6xb1lroIbsSzHrtfrjs2WD7Jh7uqdSik/NoXwl7Ztf/fR0ytqc8wo1CGOkmXLN6daV9m7Ud4Y0/wnSIh32n1oncpdipal/Axm7aV1dFxxlHIlqMSi0ahuF+a0sJwsEulVSNdLumR8H+PTVLK0ONnA5SRhkHJ1kpNJ7dvpeBOmB7GbY+W5qcBZOVSr1Q6FNbGbKZcKwP8AcMO27f8qXhrIKFlzUQDQTTxkeRWVmuxCbU7BlM1L2QiAFmU8Htd9PRuNBtLpNOLxuLZCZcaefM7jHBc7arkSLGyYnZ1FMBhEqVTq+rJLawPYipdxo+RCAbZaxrFdoIyFhcNh2Latk33BYBBTU1N6csBJwVHIVcqm2WzqiZJSYTkZMeKatyWbdvocJwvU9CxbrZa2emVyqF/YzTfmUwD+LYBrSqm3Hj33nzHAUbLSkmBmjEJho2EqNtPSkL/L+lRp4tO85y7Fji6hUAjValW/R2bXT0B3+COXK7C5mY2Pj2NychJ+v183YKFceilOp94CZFKwpaBpcfIYehPxeFy77icIRypXtpMj+wVwLpmlEuNrO1HLTKtSrlvKTp6HSWF2StuJKrVf7Car/hMAvT7h0EfJsrsR3Sy605VKRXMqaW0CW9PyzMYCPBcbHcuBbcFgEPF4XGcCOaGPi5jH0spk2OA4J4eOWq6E1+tFLBZDMpnUXdpJUOciAKATduxk9fDhQ7zzzjsoFot48OABbNvG5z73OXzoQx/S3wdaGzIuxr4FkUgEo6OjyOVyx30D7MKg5Sqz1Epttl7kjHM5opteAJt+0Fszk7ROWfVeiSKCXilzINxY2eCDhTLyeqXHsh8M/TfG6/Xq7CddtEajgXK5rHssSsXJ8cB8Ly1FANpSlDQlVgyNjIzoaqRSqYRKpaLHaMghT81mU4/nOM6Kc1jArlcjIyOIRqN6vr3k2zE0w8bUsVgMKysr+O53v4vl5WW89dZb8Hg8OHv2LC5fvqwbunBgmFycZGhEo1GMj49jY2PjpFmcRwYqzlwu1+WqMyxGb45yMWvHgW46kqz+k5/BR6W2eNjcEOkRKqVQrVb1d4dwShzvB0OvOOVN4T8qY1hy5CdvuuyZCUATcrkgnfiAMn7Kagh2UaJgKWhpubo4GNjyLZvNdik5E3TRzTg3FSoVLL8DJkUG6G5fxs23Wq32jaLioncjYZnIlTkLae1LNxzodvFN69NUoDLPEQgEdPWZeS6nZOJ+MPSKk53YOf5CKaVH/rbbbYRCoa5BTawxp1Ll0Ci/369HlUpaiwQzgrVaDdVqVe+OvPlsU0WKVKlU6lqYLvaOarWK9957D6VSCWfPnsXMzIzuiiTjU7Qs5OZZr9dRr9e7uihxc5SbJI/nI93Jhw8fYmVlZVsjGBePh6m4+Dflwvgi0N3/koqSIRj2DJDWJ9/X6/P4aNu2lj+9DDaKKZfLXcMYZQ8LKtiDcLGHXnHKLCqwPZMmdxBzxzKVI3dCKk7eRKcffpYUooznmFVJLvaHVquFXC6HcDiMRqOxjWok77+ZeZVZW2lNyqoSQrp9jJNvbGy44zMOADNuyN9N97vZbOrm4Yw7N5tN7doD6Kk4e3mGVJz0Cul98DPoKTolmgaSHDpqsHLI7/drpcfgPnctKjLeTLoJJEmHQiFNiuVr3J1isZj+YQemarWqyyuBLSFJWhPr2F0cDKVSCW+88QZGR0fxxS9+scuDaLVaelyJkzIE0LXIarUaSqVSVzEE5c3QSjAYRCAQQCaTwQ9/+EOsrq6iWCwO6L89GZD8WVrw9PbMMFir1cKdO3dw9epVrbSazSZKpVJXIkcq217ZePk35U75lkolbGxs4N69e1hYWEA2m9UbsfRAduqBsBccC8UpqUZ0y8xaWSc6A+kKtA5l3EVal5LUHgwGdeccQp5XWqsuDo5Wq4W1tTXNlOD9NeeqPw60QGXHbwlZJGHbNsrlMh48eID19XXXVd8HZBYccB5fwudzuRyWlpZ0IxeGSkwl28u7cEoSyYw9j7FtG+vr6zo5RV3B65DfpxOvOC3LwujoKHw+H8rlMprNplZ2pvvFBce2c+SUra+vIxKJIJlMdi2gfD6PWq2GVCql6StceD6fD6FQSMdrmG2v1Wq4d+8elpaWsL6+7sY3+4R6vY7vf//7WF5exkc/+lG8/PLLXXFOxppl5Q8LEagog8EgotGoZk/QHQSAUCgEpRSuXr2K69ev4+rVq1hbW9ONbl3sHlxnsveDXH+06uk1XL9+HSsrK13NPsxadnOD3K1io+LkZxaLRayurqJWq6HRaCAYDG5bo9Lz2G+4begVZyAQQCKR0OR00oNMEjStQFqPFGqtVsPy8jISiYSmHlF58iZXKhUkk0kdM1FK6d1RJhXIT1tZWcGDBw96lpW52DsajQZ+8pOf4N1334XH48Grr77aVXQgWRQA9MYmm3RYlqWbhchEBACdZb1+/Tq+853vYHFxUQ/6c7E3UEHKhB0VHWl7skLv1q1buHXr1pFcK/tLSEuYZdykLe0HQ684s9ksrl69Co/Hg3w+r838er2OXC6H1dVVWJalyyOXl5e1ywcAhUIB2WwWlUoFb7/9NkKhkJ4Z1Gw2Ua/Xsb6+rmcOPXz4EJVKBY1GA0tLS9qFbDabKBQKqNfr+jqcsn8u9ge6zwDwxhtv4C//8i+7+nPSOhgbG0MymcSDBw+wtramlZ9SCktLS7h586be9CqVCpaWljR/sFar4ec//zkWFxeRy+Vc+e0TZJ+Q3cLSxmq1ikwmg0wmg1wuNxSbUqfTwerqKu7cuaO9l0KhgPX1deTz+X1f49ArzoWFBSwtLQFAVxUIsBXzCIfDSKfTAIBMJoNarYZ4PI5oNIpSqYRMJgPbtvHBBx8A2J4FlKVbDHCTVC2P5UKTyQoX/YFt29jY2EAul8N3v/td/M3f/A0SiQQuXLiAcDiMyclJRKNRnDt3DrOzs7h27Rrm5+dRLBZ1vfmNGzd0445IJILV1VW88cYb2NjYwLvvvqv7EJh17C72BlbXBQIBrK2tYWRkRHOtHz58qJMzwxA7psUbj8cRj8eRTCaRzWbx8OFDzR3eD4ZecbIxKn8HugPTpAdxcJukJJCWQGW4UyzLFPIw7JZPGihP8jM9Ho+2Qmu1mp4+yuIEk3bCXqosUOCk0kqloluduegfeN85MYFrTdL+COkSDzovIDP/clTOQcZ8q0H+E0qpNQBlAJmBfWj/MIaDX/dp27bT/biYYYIrV1euQ4hDletAFScAKKXesG37xYF+aB9wXK97UDiu9+e4XvegcFzvz2Fft0tGdOHChYs9wlWcLly4cLFHHIXi/MYRfGY/cFyve1A4rvfnuF73oHBc78+hXvfAY5wuXLhwcdzhuuouXLhwsUe4itOFCxcu9oiBKk6l1OeVUu8rpeaVUl8b5GfvFkqpOaXUD5VSN5RS7yql/uDR8yNKqX9QSn3w6DF11Nc6LHDlejLhynWHzx1UjFMp5QVwE8BnATwA8DqAr9q2fX0gF7BLqM2Z01O2bb+plIoBuArgXwL4HQDrtm1//dGXKGXb9h8f4aUOBVy5nky4ct0Zg7Q4PwFg3rbt27ZtNwD8FYDfHODn7wq2bS/Ztv3mo9+LAG4AmMHmtX7r0WHfwqZwXLhyPalw5boDDqQ492jKzwC4L/5+8Oi5oYVS6gyAFwBcATBh2/YSsCksAONHd2WHC1euJxd7kK0r1x2wb8X5yJT/7wC+AOAZAF9VSj2z01scnhtaLpRSKgrgOwD+0LbtJ6Y7hCvXk4s9ytaV606QLe/38gPgVwD8nfj7TwD8yWOOt5/wn7X93u9B/QyjXJVSdigUsqPRqB0MBm2/3297vd6ex3u9XtuyLDsU+v/b+7bYtq5zzW+R3OTmdfMqXkRdLfkSW7aTTlOnadGgkymCybRJH87gZIBBTjHF6csAc4B5OMV5GaDAAH06mOljgVM0AxSdCZAE9UOB4qBo07hNUtdpHDuW7ViyZVkXUyLF+53c8yD9y4tbmxIpiRQl7w8QJFHcm1v73+tb//23q4qiqIqiqBaL5amWa7ey7Ydcj8BXW7nup62cnir/Fe2bGGN/D+Dv9/E5xwkLh30BHaDnchXn0ei9TrOgaAifLMuYnp6Gx+PhkymLxSIymYxuT01FUeD1euHxeDAyMoJGo4FPPvkEa2trvB0djeM4IBwFuQIdyNZYry1oK9f9EGdHqryqqj/FVvkTY2zb3w0MHA5FrowxOBwO2Gw2jIyMYHJyEqFQCBcuXIDL5YLf7+czoIj8qCGxFpIkwWq1QpIkOBwONJtNrK+vo1Ao4Pr167h9+zaWl5dx69attuRJ/V6PGXaVrbFeO8N+iPMRgBHh9ziA5f1djoEBQE/l2m7GC2MMNpsNTqcTsVgMZ8+excTEBF555RV4PB4+Nrhb0Mwiaq7rdrsBbGq7t2/f3vVajxl5Gmv2gLAf4rwKYJoxNgFgCcDfAvhPB3JVBg4TfZErEZLP58PXvvY1BAIBBAIBuFwuxGIxjIyMwOfzwWq1QlVVlEolAOAmPHX1VtUng9xoLhF96Znjk5OTMJvNOHnyJE6dOsXn5BSLRdy4cQPLy8eaR47Emm3nytkvXC4XPB4PKpUKNjY2AABOpxMWiwWlUqmrMRp7Jk5VVeuMsf8K4DcAzAB+pqrq53s9n4HBQK/k2m4xhEIhvPHGGzh16hSGhobgdrthNpthtVrRbDb5ULxqtYpGo8GnK9IQvXq9zofzkWZKozMajQafRCpJEsxmM06dOoWzZ8+2DBq7desWEokEMpkMJ066zl4t4sPAUViz2rHAB3nfyeedyWSQy+XQbDahKApkWUYymUSlUun48/Y1c0hV1V8D+PV+zmFg8NAPuQYCAYyNjWFychLhcBiKosBqtfK/02wY+gJaSYxG1JL5rqoqJEni7yETXTyGxkPTa+QDDYfDsNlsOHPmDEqlEhKJBB8QeNwwyGt2r6N69WA2m7k2SeOKw+EwYrEY/H4/FEUBAPj9flitVnz++efIZDIAOiPrgR/WZuB4QPswzszM4Ac/+AHC4TDOnTsHl8uFYrGIYrHITW06TiRKYJMULRYLZFneRoxEmDQyWJIkAOCaa61W41oNYwxOpxPnz59Ho9GAzWbDiy++iN/85jd499139zXMy8D+sZ97b7PZMDExAZfLhUgkAo/HA7/fj1AoxKfikhXSaDTwi1/8Ag8fPuw428IgTgN9hSRJsFgs8Hq9iMfjCAQCsNlsPBDTaDTAGIPJtFmboTWZ9V6j9+qlJmkDPETEpHkyxrgfNRgMol6vIxAIwG6384mpBvoD0ULQA40gpo0RACwWC0wmE2w2G5djs9mE0+lENBqF2+1GMBjk48LtdjtkWW4hTdpU6Ro6gUGcBvoGxhiGh4cRj8cxMzODEydO8BSjcrnMxzebTKZtGieBfhZn2+/kh6T3kLZK76Hz1ut1pNNpAMDQ0BACgQDm5uZw7tw5bGxsYH5+fiDmgz8t0MpbRCwWw/T0NIrFIvdFx+NxKIqCc+fOYWpqCuVyGcViEbIsY3x8HHa7HcViEbVajY+IXlpawvvvv49isYhEIoFCoYCVlZWu5GwQp4G+QlEURKNRhEIhKIoCi8WCjY2NbQ+tnuYhkh7QqmFqgwriMVoNVjwPme/AZtTVZrMhGAwiEokA2CRcgzgPFyRXt9uNaDSKXC6HfD4Pxhii0Sj8fj/OnDmDCxcu8MIIq9WKeDwOq9WKtbU15PN5vkkWCgXcv38f6XQai4uLyOVyXV+TQZwGegI9LdBsNuO5557D66+/jmg0ilqthkajAbPZDMYYqtUqJzMy2elLNM+7jXTTOek8InmKpFqv16GqKk6fPo0333wTN2/exMrKCtbX17lGLF6P4f/sHSwWCzweDzfFGWO8MKJarSIUCoExhnA4DKfTiVwuh2vXriGdTuPRo0eo1+tcnjabjZvmZrMZmUwG1WoVzWYTbrcbVqsVhUKhP+lIBgy0QztyY4zh3Llz+M53voNKpYJCoYB6vQ6LxcLN5mq1CsYYz9fUapHtfGB6JCaSo1bTFM9nMpn45zcaDUxMTOD8+fPw+/145513uKkn+l8pCGWgNzCbzfB4PJBlmW+s4XAYIyMjUFUVsVgMwKaVYLFYsLS0hPn5eSQSCXzxxRfI5/N48OABqtUqxsbG4PP5EI1GMTw8jFwux32kDocDdrsdjUbDIE4DhwstodhsNpw+fRqhUAixWAzVahW1Wg21Wg0mkwlWqxWMMVQqlZZz6JHdbgGEdtejPUardQJPUqAajQbPGQ2FQiiXyyiVSi0asUGaBwvaJGVZ5mRos9l4sYOqqkgmk5ibm+NRcYvFgmaziUqlguXlZXzxxRfcjC8Wi/w4ytTwer1QVRVWqxXhcJjnBtfrdZRKJe7r7gQGcRroCURicblc+Pa3v42ZmRlerVOtVlGpVHhdOVVviMdrczGBvZdB6kXWteY/aZW1Wg3VahVWqxUTExOwWCxIJpPI5/O6WquB/cNsNsNiscDv92NsbAwAuGZIvQkWFhaQy+UwMjKCeDwOu92OTCaDUqmE2dlZvP/++7BarXA4HHzzazabSKfTKBaL8Pl8aDabcDgcmJqa4qlH9Xod2Wy2q6oxgzgN9BxmsxmKoiAYDMJut+t2RdpJi2zXRekgQQQtflksFjidTrjdbh6V17seA/sHuWfoi7TPRqMBu90OAAgGgwiHw/B6vVyrpMDPxsYGTx0jTZTkZDKZWvzYlLUhSRI8Hg8AcLcAbZy7wSBOAz2HxWLB8PAwpqamIMsyALRU/ogRb60ZvBNJ7aUccqeUJboOKtl0u90YGxvjeYIG9get71uUhSRJvNdAo9GAJEkYGhqC1WrF6Ogo/H4/Tp8+jYsXL2J+fh7vvvsuEokE7ty5w1sNAuBpRwC4T9put8PpdIIxxk14q9UKWZYxMzPD2xUmEgnkcjmsra3p5gSLMIjTQM9BnY/EDkeiZidCS5o79e7cK2FqjxN/15ZkejweHt010FuIG6jJZILdbofdbofH44HX64Xf70cgEMDy8jJSqRQvjaWGHXQs5QMDTzRZiqqTH5si7rIsw+1288T4crm8LZNDD8f2adipUUCnKSXdLE69PEEDm6CINZU7EnlSKpKInSqIRHNae37xePG7NuFdfJ/4WWK+J+V2OhwOXLhwAX6/H5cvXz7gu/L0YafNi0xkIj3yLzscDqyurmJ+fh4fffQRms0mUqkUbt++jWKxiEKhsOvnUrtCh8MBWZZRr9exsrKCZDIJSZLgdruRSCR4QMrpdKLZbCKfz7c957EgznZaCaWZaP/WSS5gN/mC4gI3Iq76oGi1XpRcTwNsNpv8ntLxzWazhWi1iez0fr0OO/QeAvnR6NyqqvJzE3FSErWqqtzPZmB/aLc2aHMlGVmtVvj9fjgcDszNzeH+/ft49OgRFhYWulpf1K+AGn1YLBbU63Xe0EOWZTgcDmSzWR6gotLNY0+cu93IdqaZ9nVJkjA2NgaXy4Xx8XFEo1HcvXsXf/jDH3Z0GIuL0iDNJyBTNxgM8jphxhg3k8i/KRKf2MhD1DD17q+2gkjrq9S+X/yb1p9KNetixyUia8OK6D0or5JKItPpNOr1OiRJwuLiIs9qMJvN28zxnSA2yKZ2hOVyGZVKBc1mE8vLy7BYLDxlyWazwePxoNlsIplMtj3vrsTJGPsZgP8AIKGq6rmt1/wA/h+AcQAPAPxHVVU32p3jMLBblFZvIVBrsdHRUbz00kt4/vnn8d577+Hjjz/eNdK2mzN50NAPudpsNoRCIYTDYXg8HjgcDlQqFZ7kTiQlkqTFYuG5dUSqorxEzVK856I/tFNZiBseEafNZuNpLNq2dkcFR3HNUvpQpVLh2uDs7CyAJ5sfaYTihtYJ7HY73G43TCYTisUiJ85KpYL19XWoqgqXywWn08n7ue4m807GA/8cwCua134I4Leqqk4D+O3W7wMF7YIjMMagKAoikQhGR0cxNTWFs2fP4tKlS3jhhRdw6tQpjI6Owu12Q1VVKIqCs2fPtkSE9UAmQTQaxfj4OE9zGGD8HD2WK7kw6Evr+tC2j6Nj6Ls2f1Pv53aviSko7d6r1yREm2hvtVp5gv4Rws9xBNcsQXSt6Gn87QKLeqDNmKwJvWdC9G/TeXcb07Krxqmq6h/Y5qB3Ea8BeGnr57cA/B7AP+76X/QR7bRKs9mM6enRdW8BAAAdgUlEQVRpxONxBINBhEIhDA0N4eLFiy3lXRaLBblcDqOjo/je976HhYUFvPXWW7pJsnSjXS4XvvrVryIajeLKlSv49NNP+/Gv7gn9kqtInOJDSxVDoh+aQPdTjzxFP6ZW8xT/1g56gUGROEUt02KxwO12I5/P876eRwFHdc0S2rUU1Ns8O4k/UOScSnotFkvLpq6qKu/vSi3mepWOFFZVdWXrwlcYY0M7XPihjBulm0JjGMh/ZrPZWmbaEHkGg0HIsszLsADwoEAsFkOpVGq7eMxmMxwOB5/GGAgE4HA4+vnvHhQOVK5EjnT/RdISd/itz9Mls/2CouQiaeudu9210HVShZMYwDhi6Ei2h7Ve20F0wdDv3fqbae3T+icyBdCy3rtBz4ND6iGNG3W73XC5XAiHw5ienobD4UA8HofL5cLMzAyvma5UKjCZTCiVStz/0Wg0eDDD7/cjEom03GwtqB8gJekGg0Hemv+4ohO5yrKMeDyOeDwOp9PZEsneOg4AeLSd6tfJlwW091dqA0P0M2kQev5OrZahzfcjk65cLvMBcJVKBY1GA5FIBBMTE1hbW0Mqler+hh0RHNZ6FaGVuRgY3MuGSlH14eFhjI2NYWNjA7Ozs0in07h58yZKpRJqtRrK5TLXSncj570S52PGWHRr54oCSOzxPAcOWhjkEA6FQpiYmICiKDhx4gQ8Hg9Onz6NSCSCTCaDVCqFarXKO/WUSiVe5kUak6IoUBRFN+eQKhMikQgnTKfTyf1iRywae6BypZJFatrQzrwiE5kCBGK0Hdi+kHa6p1rtktDOdwqgRRsWzX+6HmBzGqLP5+MVKkcQA7tmO4WobbaTsd6zQZanx+NBLBaDLMtYW1vjmyXJWqxd3w17Jc7LAN4E8OOt77/a43nagh5ibf6dHsiUcjgcOH/+PEKhEE6cOME1zEAg0DK0KZvNcu2G0hKo1KtUKqFSqXDTe2VlBZ9++inm5uZ4Xhf19xsaGkI0GsXk5CReffVVWCwW/OUvf8Hjx4+RTCYxPDyMYrGIjY2No0KgPZOrSFhiF6Kd3i+mDImkJr6H0lPod+3xBPEcJGtKsqdxHmL5JxE4uWHOnz/fkjx9BNHzNdtPtJO5+DcCtTBMpVJYWVlBvV7H0NAQJEmCyWRCtVqFLMsIBALIZrMtzVzaoZN0pF9i06kcZIw9AvA/sHnz32aM/RcADwH8Tcf/cfvPAYCWHYUe8N1AtcQejwfPPfccTp48ifPnz+PUqVN8RGytVkM6nea1rJlMpqXnI5mHNDaWOqmsra3h6tWrWFpaQrFY5HlhsiwjFovhwoULOH36NF5++WUUi0W89957+Oijj2C32xEOh5FMJpFOpweOOPslV1Gb69ZXJRInTbAUcz61mmOnUVZyGdB56FnT83OazWbY7XacPHkSLpcLt2/f3sNd6C/6JdvDhMgT2udK+75qtYpSqYRMJoNEIgFZluH3+7nMqROW1+vlwaHdeKeTqPobbf70b3c7thtoiZN8UHoRV5qf7XK54PV64Xa7MTw8DLfbjWeeeYb36stkMjwnkFRx0jC0oM9yu91cMy2Xy7zJgKIoYGyzSQClNvj9fgSDQdRqNVy+fBm5XA4LCwsolUq8EmJQh331Q650DyjhXQsxcX23dCTxb3qkuRtEq4WOFxefWGpJzwkdR7+LGukgo19rdhCwU/YMucxqtRpSqRTK5TISiQScTifi8Tiq1SoP/lKeZ6PRQLFY3NVcH4jKIb1F0M48N5lMiEajCIfDmJqawpkzZxCNRvHlL3+ZD/5qNBooFApYXV1tefAp3YgK/mlRi3NnQqEQT5TNZrNwOBx49tlnIUkSvvvd7/LjGo0GUqkUkskkbty4gZ/85CfY2NjgBFwulyFJEsrl8sBpm/1Cs9nkkyLFnEmx4zrwZPCaVovYb2SdPkubC6hNjSIznfpwisUOdG303BwF4nya0G5tkSuNGncsLy9jY2MDyWQSgUCAu178fj/8fj9PfKe5RLsVvAwEce4Em80Gl8sFSZL4MK0TJ04gFArxXEyPxwOr1dpSKge05vppE7G15qMYuSN/GGkZog8MANbW1pDL5bggUqkUb0Pm9Xp5yoPZbEYymeQpD4OqffYKVqsVgUAAgUCAj3UVLQoCyYSqhQ4qFUlEu2II8RkQ/el0HUS6TqcTiqIY7eUGHGK3o2AwCJvNhlwuh3K5zIPGlFYmbpJknpPitW9TvR/Yyd9F2mQwGMSlS5fg8/ng8/ngcrm4xsgY46VT4oAmt9ute04xZUVcpM1mE8VikSfEUhs0xhgvz8rn83j77bdx7do1/vmBQACXLl1CIBDAV77yFYTDYd5o9dq1a3j33XdRqVSwuLjYk/s3qAiFQvjWt76FeDwOv9/PH0gKygBP/NMESg87CI0TeLJ5assoKaJKGyLVMVMHJ6vVCgD896mpKdTrdYTD4aOYLXHsoFd2CwDDw8N44YUX4HA4oCgKVFXFtWvXkMlkEA6Hcf78eb4hVioVPHjwALlcDl6vFy6XC8lksqVLUzscCnHSwtBqgMD2KgGv14toNIpIJILJyUneMMJms6FQKCCbzfJBS6JPVJblbV279RzHWtA5xGABaSLkZF5fX+cVRORzDYfDiEQiOHHiBCKRCI/OLS8vQ1GUlrEQTwskSUIoFEIoFIIkSbruF1H22tLMg4BedJ0gPoei5SHmm9IGS53riVANDAbE5tM0aSAcDsPhcMDhcKBer3MXnd1u5wEgUpBKpRIKhQJ3w1CWTa/yOPcMk8mEQCAAp9PJ51fLsgyv18t3fnpQZVmGz+fD8PAw/+cLhQIKhcK2SJq27T6RpljpoRfZ1ZbrUXoLReMo6maxWDiBf//738drr73GzXhJkriJ3mg0eAdpxjYn8126dAm1Wg23bt3q120eCNhsNvh8Pni9Xq71AeCaHk23zGQy20i1XWrJfiH6VqnjuDboo5f6lMvlUKlUeODPwOGCNjhSomZmZjA9PQ2v14tYLAbGGB8GOL7VO4KIkrIpzGYzt0olSeqqKqzvxMkYg9PphN/vx8jICKanp+FyuRCJRLgvkWYqO51OnvUvRqkrlQrq9TpPCxK1Bto5xARqWija+me90joxmCDWrNJuRhPyAPBywkqlwhPp8/k8r22mWufJycmOkmqPG0wmE28gK6Z4iClA9XqdV2xQaaYWB0VUovUAoIU0tUSt9X+XSiVeYWJgcEBjMaampnDp0iUuy0ajgWw2i2aziWAwCIfDgXQ6jfX1dQBPyjBtNhtfm53kjBP6Spx2ux1nz57FqVOnMDQ0hGAwiKGhIe6PFE0k0gYAtDhwAfCHnbRDcVGST4oWp+hTE6tXtGkvWnNRa8ZRxA14srgogZ6i7KqqcoGQJvo0jlygzY42PsqPFdPB6OGm+6atCNFLcKb3ac1pgl4+n/Y89KyQdSO2rhPHAovXQRsAlWQaeIJeWAbapHb6IkVGkiRekfalL30JQ0NDiMViPJPFbrcjl8vh6tWr2NjY4CW0olnu9Xq5L5uKYvL5fEfllkCfidPlcuHFF1/ExYsXEY/HObmIxKVtcktlUC0XvUWApEFQxFqb0iIuCiI/ceaNaNrTe0WfCS0gOkbMSRQXpJhzSucjbfQoddU5KFAxgsvl4g8maZVEWHTPtL0VtdaASHj0PBC0myBBG73X1rM3Go0WrZc+n3J+xa5NdA1OpxOyLBtRdQ12Sj7f6/m0MQ/iAjKlJUlCIBBAKBTCN77xDUxOTiKVSiGdTvOeCMViER988AEWFxe3baJutxsjIyOQZZln5Aw0cdZqNayuruLhw4eo1Wq8ntxsNkOWZf4g040TyU97M8Xfxdpm7fHigiSSFv2ZotYqmufiedotTC2Bat9DJt76+vpTZarbbDYepRQ1N71MBq1LRM+Noof9LFKRrLXRU9HfqQ0s9SJN6qiDZHpQGqd4PvpOz4hYIUjdzcgkbzQasNlsyOfzWFpawsrKCp9oqYWY1yu6aWgj7wR9Jc5CoYA//vGPuH//PrxeLyKRCK/4iUQiPJhgs9m4tkYBGJHw9BYh/Z0aStD7RBKl10UtU5uORCZlrVZrqTgSNRfRtKN0KHGWs/h6MpnE559/jnK53M9bfajw+Xw4efIk4vF4iyxEU526D5FFQQ+4OBpBG90U7/9eIZK01i1AGzi5F0jDEZ8hgzxbsdPmtldo5UOuOuofMT4+jm9+85twOBzI5XJIpVKIRqMIBoO4evUq3n77bR5r2A0kW4qfdNpirq/E2Ww2ebE9mW2SJKFYLAIAbyQqTqUjBy418tDTOkXNk1RtrUBJ5ReP1aapNJtNntZEOX20sEWtlCAueDIjqcsSmfqJRALpdPqpIk7yBYryaqed61kSevITzXY97X6n38Vj6G8igYoLVZsSZWichw+SAWXZKIoCh8PBNUxVVXkzHarm2229iTIXC2c63QT6HrlQVRXpdBrZbBYrKyu4efMmH5JEO77FYuE16NTOi4rwyTEsppIATx5qiqrTwqM+e5SHqaoq97dRiV29XucVA6VSqeV3seJH1Ez0zJNmc3OkKGmrlGqTSCSOavPbPYE0drEhNI1/1WZBmEwmyLKMZrOp23pOa9ZrSZY+T88NQO+jTY0CUvQzaTLkRqFnT7w2sW7dQH+hXTNTU1N46aWXYLPZuHbo8/nAGMOHH36IGzduIJ1Od1yhx9hm2iMpaO16KujhUIiT6reJuAhiAMfn88Hj8cDtdvPu7FRCRYO/SKujY4Ht882p25GoTRYKBZTLZT7xTuzDSRojEateqeROs0tyudy2mueDNmUGHaJrQ9Tw9LRO0c0h3lPRtN9JWxW103a+tnY+aq0LBtiehC8GFgwC3Rv2G3mnTczn82FiYgK1Wg3ZbJZPwTSbzVhfX8fNmze7PjdlwIjtCTs6roOLHgHwfwBEADQB/FRV1f/N9jA1j8iLHk6aKig+kGK1DvXGzOfzsFgsWF1d5aMwtH7Kdg+1uED0mkqQVkSLh4hb7Pau1WrF1+ka6MYXi0WuxZKWS1UKg9TH8SDlqkU2m8X9+/cRiURafEYiyWnTkEjmFPXW64xFoACONudWa+prnytguxYjunvEwJBoyh8l9FKunUBsokNFK+Q3TqfTO47c1YPFYsEzzzyDWCyG6elpqOpm9VYoFEKpVMKHH36I1dVV3L17d9uxeoFIretFlmWeatbVdXXwnjqA/66q6ieMMTeAa4yxfwXwd9icmvdjxtgPsTk1b8fhT6QBij5LkTjFnYlItl6vY319vSUqrn2oxYdfXIi0GMThTKThiP4ssfMNfQaRsyRJLRF/4ElZpngOStInrZbImXy6zWZzoIgTByhXLXK5HK+gEs11glYLBfSJE2jf6VtvwxS1Q9Jg2/lFxXMTyWpLL3sR+OgDeibXTkBrQZIknuqjKApkWYaqqkilUt1pdhYLTp48iZmZGT7Hi869sbGBv/71r201zXaWA/2NfPHa8uyOrmu3N6ibA55oyFOOMTYLYBj7mJonaiHtfEjiAyz+8/SAa7sctRtrIRIe0NpSTPu5FAiiKBsAPqxeex4RRLDAk+FPZO6XSiWk0+mB0156IVeCy+XiUU5x/K/YZIMWmJjLCWyfEbRTIK+dM38nS0SUvx6pizLW01wHHb2UayewWq3weDw8JU2cvEDWXCfaPGObDcMdDgd8Ph9CoRBkWYbdbkcmk8Gf//xnJJNJPoO9HUTC1HtWSOPsNt+6K5pljI0DeBbAx+hiIqIWRCqVSqXtQ0nqvkiStGBIGxRL5rSaiVhZIuZqaqEN9FQqFa4Zi7mXonapNRNpoQJPNBU6njpPDxpxijgouRICgQDOnj2L8fFxnuxOMqdMBOo+Rb9rU7+0m554/0TNVC+ivpNGKrpVSL7aTVh0KRxRrRPAwcu1E9DkAxpFQVVj5I+klm6UjtYOVEfu8XgQj8cxOTnJ/ZGrq6v45S9/ibW1tR0DQVotU/t5JpMJHo8HPp+v7SDGduiYOBljLgDvAPgHVVWzne7CbJdxo+1unhgYEN9L2qCodehpnNTbcesaALRqukD7IV0UFabgjhh4os/SBi20xEkkQRrooC6+XsiVmraICfDA9qAMyfLx48eoVCpQFAV2u52/VyQ+vQCD3v3f5Zpb3kfZF2LWBJXparVZuh5y/Qy6D7RX63U3UKcyVVWRzWYhSRInTlJKOtmMrFYrotEo/H4/fD4fHA4H8vk81tfXkUgkeIB3ryDesNlsvIlxN+jo3YwxCZtC+IWqqu9uvdzR1Dx1j+NGxRxJUejaxdTugejWvNLzhWiFq70OraNZ7/hucsP6jV7JdWhoCOfPn8fY2FjLmF/RhUGklUwm8atf/QqpVAqvvfYaLly4wP3QomZCJCVWd2irjbauS/dnsUyXfrfZbCiVSrh//z6q1So8Hg9vuEw5vFrIsgxFUfgAsEGU7WGsV0I2m+UBYHG2EwCeadLJPfP7/Xj99dcxPj6O6elphMNh/Pa3v8Xly5extraGQqGw6zn0ynNFMMbg9/sRjUbhdDq7+C87i6ozAP8CYFZV1X8W/tTzqXntVGwD+0cv5Wq1WvlcerHsFdheO16r1bC8vIxEIsGDSmJPAXF0hbgZaf4Xfm7tAhE1V9HCECPoxWKR5+2K59LTjqiJxKBqm4e5XgHwXGjyYWutOq2bRAsK0LpcLsTjcYyOjvJ5QPl8HouLi7zEcr+gzJ6eBIcAvAjgPwO4wRj7dOu1f8Ixm5r3FKJnchWzE6j5Cj2kWhdHqVTC7du38eDBA0xPT6NcLiMej2N8fJwvNDrWZDLxQVp67pl2kXsAvKEMvV4sFrG6uor19XVcuXIFpVIJfr8fkUiEH0ckLVo4sVgMMzMzWF5eRi6XG8QeBAOxXoPBIF599VWEw2E+NO3evXuYnZ1FKpXCvXv3dP2Tw8PDuHjxIsbHx3Hx4kWEw2HMz8/jiy++wJ07d7C8vMw3uf2Cii8cDsfBB4dUVb0CoJ3de+ym5j0t6KVcRV8ikRylf4hVXcCm+ba6uopHjx5hbm6ON32hudekDVCwkPxnYjCHINayk4kozkcX+wlUq1Ukk0msrq5ibm6OTxOg4/W+M8agKAri8ThKpdJARtsHZb263W48//zzmJychNfrhd1ux8cff8xbvz148ECXOBVFwTPPPIOxsTGMjIxAURTcvHkTDx8+xOPHj/ecoaIXGGJbKY60oXeDp69ZpIG+gMiLHkwaHdLuAa3X67h9+zYymQxu376Njz76qKW1H+XZUtCOCiH0/Juitil2nRf90sViEZlMBtlsFrOzs2CMbWsKIWqaZMb7fD6cOHEC2Wy2o4DUUYYkSYhEIi3FIuQ6EeePi7588WeKU9DIiomJCRQKBbhcLszOzvICkWazyWvQT58+jWeffRY+nw+ZTAaZTAbXrl3DtWvXMD8/fyBuO3HDpRTEbmEQp4EDhxjppr6ku80kr9fruH79Oj777LO2QT8KGlHeLBVP6AXmgCeLQiym0LtWVVWhKArPCRS1WQpwEAGHQiGcOXMGjx8/PvbEabPZMDExwQmTGnfXajUUCoWWqZBiOhl90XFOpxOBQADT09M8c+J3v/sdL2lWVRWhUAijo6M4d+4cXnjhBTDGsLS0hPX1dfzpT3/C73//+65qyXcDWUHiZtANDOI0cOBIJpO4desWqtUqLly4wP2Tet2HRHSSpkL+Uu20TL3UNdKSyDTfaXE0m5stBYlgxZxOOr/JtDnMK5PJPBXD90ymzc73Yt4yESLdT/E7bVL1ep37isvlMi+zpFSifD4PRVEAbG5EAHDixAnE43H4fD4UCgVUKhXcvXsXa2trvJuaGPzbD4GSVlyr1ZDJZJBMJruWp0GcBg4cN2/exNLSEr7+9a/j5Zdf5pUZlCdJpLSXXEgiTMoL3A2dZmao6maDlrW1NV6hQiRAQSqLxYJkMol79+5hdXV1YCPrBwWLxYJQKASfz8eHJ9LIbOqET9oa9ZUgUqLg4NraGu81sb6+jtXVVRSLRYyNjcFkMmFsbAwejwfRaBShUAgmkwkLCwtYXV3FO++8g0ePHuHhw4ctbSj3al4TaCMtFouYn59HLpfrvoZ+z59uwEAb0CTBtbU1LC8vQ1XVlo5SZPoWi0Ue7OkWB13Ro6qbfQby+TwYY1zTIq2TNKpsNou1tTXkcrljT5x0TygvUywAEHM0aUPUljaTWU/TJnO5HG/xSDmzgUAAiqLA5XJBkiSUy2VkMhk8fvwYiUQCyWSSb5AHJW8xzYwCj0ZwyMChgwIHn332GX70ox8hFovhjTfewPT0NDfbk8kk7t+/j7t373aUzKytINLzbYrQvrab5tloNLC4uIibN29iamqKm5CyLKNSqeDBgwfI5XJ4//338etf/xr5fH4QU5EOFLlcDh988AEPzIkzuohEKevBbrfz5h70XVGUlsIDq9XKNVfqrev1emG1WpFKpfDgwQOsrKzg1q1byGQyPNtBHNZ4EFNGqcDBYrHgueeew5kzZ3Dr1i1cuXKl43MYxGngwEER1WQyiatXryIWi+GVV15p0U7K5TL3X3Wak9euakzr39zpurTnEwk1l8thfX0d0Wi0JWHbZDJxX9jy8jIWFha6uR1HFvV6fceOXuQDtVqtvK7carXC4XDAarVykx3YvNdut5sTJmmc1JC8WCwikUhgcXERd+7cQalUQjab3UaUB6F1ku8b2Mw3jcVicLlcXZ3DIE4DPUO9Xkcmk4Hb7ebVP9SD1eFwcBONTHjSaMS2cgQyrbSaZzt0ooVqfa3pdBorKysYGxvjgSer1YpisYjZ2VnMzc1haWlpP7fkWIECajTxIJ/Pb+vHSXIS++9SZRClqzHGkM1mkcvlkM1msbGx0TJB4KBBQSFJknD9+nWUy2UsLy93dQ6DOA30DNRRn+bCkH+MEttdLhccDgf3jZHvTNQItDioMlxtyhM1paCoL5V+Uine3Nwcrl+/jvX19X197nEDJbEfpZlajUaDk/ydO3e49dMNDOI00HNUq1XcvXuXjzsxm81IpVJYXFzEwsICTwXR64jVa4hETHmK4rRDseHHQQekDBwuqtUq5ubmkE6nkUjo9jxpC4M4DfQcpVIJV65cwfz8PMrlMiqVCp9KmM/neeK5WIXSD2hJsFwuI5fLoVQqtYyHpnlUBmkeL9DoDZPJ1HWgzyBOA30BDcYrlUot32ny6CBA7CpP/tR+ErmB/mOvUXrWz4eWMbYGoADgKDqKgtj/dY+pqho6iIsZJBhyNeQ6gOipXPtKnADAGPuLqqr/pq8fegA4qtfdLxzV+3NUr7tfOKr3p9fXfby7FBgwYMBAD2AQpwEDBgx0icMgzp8ewmceBI7qdfcLR/X+HNXr7heO6v3p6XX33cdpwIABA0cdhqluwIABA12ir8TJGHuFMXaHMXaPMfbDfn52p2CMjTDGfscYm2WMfc4Y+29br/sZY//KGPti67vvsK91UGDI9XjCkOsOn9svU50xZgZwF8C/A/AIwFUAb6iqeqsvF9Ah2ObM6aiqqp8wxtwArgF4HcDfAUipqvrjrYfIp6rqPx7ipQ4EDLkeTxhy3Rn91DifB3BPVdV5VVWrAP4vgNf6+PkdQVXVFVVVP9n6OQdgFsAwNq/1ra23vYVN4Rgw5HpcYch1B/STOIcBLAq/P9p6bWDBGBsH8CyAjwGEVVVdATaFBWDo8K5soGDI9XjCkOsO6Cdx6jVQHNiQPmPMBeAdAP+gqmr2sK9ngGHI9XjCkOsO6CdxPgIwIvweB9Bd99A+gTEmYVMIv1BV9d2tlx9v+VPIr9JdH6rjC0OuxxOGXHdAP4nzKoBpxtgEY8wK4G8BXO7j53cEttnd9l8AzKqq+s/Cny4DeHPr5zcB/Krf1zagMOR6PGHIdafP7XN3pH8P4H8BMAP4maqq/7NvH94hGGNfA/ABgBsAqJ/YP2HTb/I2gFEADwH8jaqqqUO5yAGDIdfjCUOuO3yuUTlkwIABA93BqBwyYMCAgS5hEKcBAwYMdAmDOA0YMGCgSxjEacCAAQNdwiBOAwYMGOgSBnEaMGDAQJcwiNOAAQMGuoRBnAYMGDDQJf4/CORZoMLIUToAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 9 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot\n",
    "trainX = X_train.reshape(-1,28,28)\n",
    "print(trainX.shape)\n",
    "#lot first few images\n",
    "for i in range(9):\n",
    "    # define subplot\n",
    "    pyplot.subplot(330 + 1 + i)\n",
    "    # plot raw pixel data\n",
    "    pyplot.imshow(trainX[i], cmap=pyplot.get_cmap('gray'))\n",
    "# show the figure\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784) (60000,)\n",
      "(10000, 784) (10000,)\n",
      "float32 uint8\n",
      "0.2829032\n",
      "0.29028687\n"
     ]
    }
   ],
   "source": [
    "train_X = X_train.astype('float32')/255.0\n",
    "test_X = X_test.astype('float32')/255.0\n",
    "print(train_X.shape,y_train.shape)\n",
    "print(test_X.shape,y_test.shape)\n",
    "print(test_X.dtype,y_test.dtype)\n",
    "print(np.mean(train_X[0:1000,:]))\n",
    "print(np.mean(test_X[0:1000,:]))\n",
    "train_y = y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(np.unique(train_y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w:  [[ 7.13872271e-02 -6.25326773e-02 -7.78654011e-02 ... -6.73204510e-02\n",
      "  -2.25992186e-01 -3.96112186e-01]\n",
      " [-1.39405080e-05 -3.38717194e-06 -1.81418678e-05 ... -1.26165734e-06\n",
      "  -4.97543204e-06 -1.66549973e-06]\n",
      " [ 1.27561238e-04 -2.00792089e-06 -4.44557588e-06 ... -7.95648528e-06\n",
      "  -5.68093074e-05 -2.09169948e-05]\n",
      " ...\n",
      " [-1.37307415e-02 -1.60786537e-03  1.83036097e-02 ... -4.47168080e-03\n",
      "  -1.36509944e-02 -2.70972393e-03]\n",
      " [-4.61674182e-03 -3.50748604e-04  3.85299716e-03 ... -1.58283627e-03\n",
      "  -5.53167732e-03 -1.48483069e-04]\n",
      " [-2.10709303e-04 -1.07533380e-04  7.93387181e-04 ... -1.84603646e-04\n",
      "  -8.60373296e-04  1.02873073e-04]]\n",
      "[2.2761396889046304, 0.7964752797732344, 0.6907536377516504, 0.6407071871107026, 0.615104492285584, 0.5934714313862737, 0.5762885729260426, 0.5647093666411506, 0.5580982571652005, 0.5463751608858817]\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "batchsize = 50\n",
    "epochs = 5\n",
    "shuffle = True\n",
    "alpha = 0.01\n",
    "reg = 1e-3\n",
    "gamma = 0.8\n",
    "\n",
    "w = np.zeros([train_X.shape[1]+1,len(np.unique(train_y))])   \n",
    "w_history = batch_gradient_descent_softmax(w,train_X,train_y,epochs,batchsize,\n",
    "                                                shuffle,reg,alpha,gamma)\n",
    "w = w_history[-1]\n",
    "print(\"w: \",w)\n",
    "X,y = train_X[0:1000,:],train_y[0:1000]\n",
    "loss_history = compute_loss_history(w_history,X,y,reg)\n",
    "print(loss_history[:-1:len(loss_history)//10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x233bdeb1490>]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAey0lEQVR4nO3deZzcdZ3n8denq6rvTrqT7pCjcxPAcOSwjRyuBAQJoLK4rJOsg8fKZnFgF2dcHdAdXdl1dJ15eKAoZhSZUYHHjBJhMSCIIChnBxKSkISEnJ1u6O6cnU7f/dk/6tdJpc9Kurqr61fv5+PRj676/r5V9f1qeNe3P7/L3B0REQmvnHQPQERERpaCXkQk5BT0IiIhp6AXEQk5Bb2ISMhF0z2A/pSXl/usWbPSPQwRkYyxdu3aRnev6G/bmAz6WbNmUV1dne5hiIhkDDPbPdA2lW5EREJOQS8iEnIKehGRkFPQi4iEnIJeRCTkFPQiIiGnoBcRCblQBf1dT23jj282pHsYIiJjypBBb2bTzexpM9tsZpvM7LZ++nzczF4Pfp43swUJ23aZ2QYzW2dmI3oW1D1/fIvnFPQiIidJ5szYTuDz7v6qmZUAa83sSXd/I6HPTuBSdz9oZlcDq4D3Jmy/zN0bUzfs/sUiOXR0dY/0x4iIZJQhg97d64C64HGTmW0GpgFvJPR5PuElLwKVKR5nUmKRHNq7dMcsEZFEp1SjN7NZwCLgpUG6fQZ4LOG5A0+Y2VozWznIe680s2ozq25oOL3yS27EtKIXEekl6YuamVkx8Gvgc+5+ZIA+lxEP+vclNF/i7rVmNgl40sy2uPuzvV/r7quIl3yoqqo6rWV5LKrSjYhIb0mt6M0sRjzkf+nuDw3Q5wLgJ8B17r6/p93da4Pf9cBqYMlwBz2QWCSHTpVuREROksxRNwb8FNjs7t8eoM8M4CHgRnd/M6G9KNiBi5kVAR8ENqZi4P2J1+i1ohcRSZRM6eYS4EZgg5mtC9q+BMwAcPd7gK8AE4Efxr8X6HT3KuAMYHXQFgXud/fHUzqDBKrRi4j0lcxRN38CbIg+NwE39dO+A1jQ9xUjQ4dXioj0FaozY2ORHDo6VaMXEUkUrqCPqkYvItJbqIJeNXoRkb5CFfSq0YuI9BXCoFeNXkQkUeiCvr1TK3oRkUShCvrcqGr0IiK9hSroVaMXEekrhEGvGr2ISKLQBb2OoxcROVmogr7nOHp3repFRHqEKuhjkRzcoatbQS8i0iNcQR+NT0d1ehGRE8IV9JH4dFSnFxE5IVRBnxuJX01Zh1iKiJwQqqDvWdEr6EVETghn0Oua9CIix4Ur6KOq0YuI9BaqoFeNXkSkr1AFvWr0IiJ9DRn0ZjbdzJ42s81mtsnMbuunj5nZXWa23cxeN7PFCduWmdnWYNvtqZ5AIgW9iEhfyazoO4HPu/u7gAuBW8xsfq8+VwPzgp+VwI8AzCwC3B1snw+s6Oe1KXP8OHrtjBUROW7IoHf3Ond/NXjcBGwGpvXqdh3wLx73IlBqZlOAJcB2d9/h7u3Ag0HfEZEbVY1eRKS3U6rRm9ksYBHwUq9N04C9Cc9rgraB2vt775VmVm1m1Q0NDacyrOOiOSrdiIj0lnTQm1kx8Gvgc+5+pPfmfl7ig7T3bXRf5e5V7l5VUVGR7LBOohq9iEhf0WQ6mVmMeMj/0t0f6qdLDTA94XklUAvkDtA+InpKN226b6yIyHHJHHVjwE+Bze7+7QG6PQJ8Ijj65kLgsLvXAa8A88xstpnlAsuDviMiNxIB0A3CRUQSJLOivwS4EdhgZuuCti8BMwDc/R5gDXANsB04Bnw62NZpZrcCvwMiwL3uvimlM0iQH4t/b2lFLyJywpBB7+5/ov9ae2IfB24ZYNsa4l8EIy4vFl/Rt3Z0jcbHiYhkhFCdGasVvYhIX6EK+txIDmZa0YuIJApV0JsZ+dGIgl5EJEGogh7i5ZvWDpVuRER6hC7o87SiFxE5SeiCPj+Wo52xIiIJQhj0WtGLiCQKXdDnxSK0akUvInJc6II+P5qjFb2ISILwBX0sQpuCXkTkuBAGvQ6vFBFJFMKgj9DWqRW9iEiP0AV9XlQrehGRRKEL+vxYhFat6EVEjgtn0GtnrIjIceEL+qB0E79EvoiIhC7oe24+ossgiIjEhS7o8xX0IiInCWHQB3eZUp1eRARI4p6xZnYv8CGg3t3P62f7F4CPJ7zfu4AKdz9gZruAJqAL6HT3qlQNfCD50Z77xmpFLyICya3o7wOWDbTR3f/B3Re6+0LgDuCP7n4goctlwfYRD3mAvGBFr0MsRUTihgx6d38WODBUv8AK4IFhjWiYTqzoFfQiIpDCGr2ZFRJf+f86odmBJ8xsrZmtHOL1K82s2syqGxoaTnscPTtjVboREYlL5c7YDwN/7lW2ucTdFwNXA7eY2fsHerG7r3L3KnevqqioOO1BHN8Zq9KNiAiQ2qBfTq+yjbvXBr/rgdXAkhR+Xr+0ohcROVlKgt7MxgOXAg8ntBWZWUnPY+CDwMZUfN5gelb0qtGLiMQlc3jlA8BSoNzMaoCvAjEAd78n6HY98IS7Nye89AxgtZn1fM797v546obev54VfYuCXkQESCLo3X1FEn3uI34YZmLbDmDB6Q7sdBXlxqfU3NY52h8tIjImhe7M2KI8Bb2ISKLQBX1uNIfcSA7N7SrdiIhACIMeoCgvohW9iEgglEFfmBvlqIJeRAQIadAX50W1ohcRCYQy6OOlG9XoRUQgtEGv0o2ISI9QBr1KNyIiJ4Qy6IvyohzT4ZUiIkBYgz43otKNiEggnEEflG7cPd1DERFJu9AGfWe309apSxWLiIQy6It1vRsRkeNCGfQnLmymHbIiIqEM+uK8+DXptUNWRCSkQV8YXJP+WLuCXkQklEFfnB8P+iat6EVEwhn04wtiABxp6UjzSERE0i+UQV9WmAvAweb2NI9ERCT9hgx6M7vXzOrNbOMA25ea2WEzWxf8fCVh2zIz22pm283s9lQOfDDjgtLNIa3oRUSSWtHfBywbos9z7r4w+LkTwMwiwN3A1cB8YIWZzR/OYJMVjeQwLj/KoWMKehGRIYPe3Z8FDpzGey8Btrv7DndvBx4ErjuN9zktpYW5HDqm0o2ISKpq9BeZ2Xoze8zMzg3apgF7E/rUBG2jorQwptKNiAgQTcF7vArMdPejZnYN8BtgHmD99B3wKmNmthJYCTBjxoxhD6q0MJeDKt2IiAx/Re/uR9z9aPB4DRAzs3LiK/jpCV0rgdpB3meVu1e5e1VFRcVwh0VpQYzDKt2IiAw/6M1ssplZ8HhJ8J77gVeAeWY228xygeXAI8P9vGSVqXQjIgIkUboxsweApUC5mdUAXwViAO5+D3AD8Fkz6wRagOUevxB8p5ndCvwOiAD3uvumEZlFP8YX5nK4pYOubieS018VSUQkOwwZ9O6+YojtPwB+MMC2NcCa0xva8JQWxHCHptYOSoMTqEREslEoz4wFKCuKXwZBx9KLSLYLbdCXFgSXQdAOWRHJcqEN+vGFwYpeO2RFJMuFNuh7Lmyms2NFJNuFNuhLC1SjFxGBEAf9uIIYZgp6EZHQBn0kxxiXH1PpRkSyXmiDHmBiUS6NuvmIiGS5UAd9eUkejU1t6R6GiEhahTroK0ryaDiqoBeR7BbuoC/Oo0ErehHJcuEO+pI8mlo7ae3oSvdQRETSJtxBX5wHoFW9iGS1cAd9SRD0qtOLSBbLjqDXil5EspiCXkQk5EId9BOLconkGHWHW9I9FBGRtAl10EcjOUwtzafmoIJeRLJXqIMeYHpZIXsPHEv3MERE0ib0QV9ZVsBerehFJIsNGfRmdq+Z1ZvZxgG2f9zMXg9+njezBQnbdpnZBjNbZ2bVqRx4sqaXFdLQ1KaTpkQkayWzor8PWDbI9p3Ape5+AfC/gVW9tl/m7gvdver0hjg80ycUAqhOLyJZa8igd/dngQODbH/e3Q8GT18EKlM0tpSoLCsAYOO+w2keiYhIeqS6Rv8Z4LGE5w48YWZrzWzlYC80s5VmVm1m1Q0NDSkb0JyKYgB2Njan7D1FRDJJNFVvZGaXEQ/69yU0X+LutWY2CXjSzLYEfyH04e6rCMo+VVVVnqpxTSjKpbQwRqMugyAiWSolK3ozuwD4CXCdu+/vaXf32uB3PbAaWJKKzztV00oL2HdINXoRyU7DDnozmwE8BNzo7m8mtBeZWUnPY+CDQL9H7oy02eVFbHvnaDo+WkQk7YYs3ZjZA8BSoNzMaoCvAjEAd78H+AowEfihmQF0BkfYnAGsDtqiwP3u/vgIzGFI75oyjkdfr+NoWyfFeSmrVomIZIQhU8/dVwyx/Sbgpn7adwAL+r5i9M0NdsjuaDjKBZWlaR6NiMjoCv2ZsQBnTioC4K0GlW9EJPtkRdDPnFhENMfYXq+gF5HskxVBH4vkMGNiIW/V61h6Eck+WRH0AGdWFLNdpRsRyUJZE/RzJxWzq7GZjq7udA9FRGRUZU3QnzO5hM5u1/H0IpJ1siboF88oA+DDP/hTmkciIjK6siboe65i2dWdssvoiIhkhKw5TdTMuHL+GezeryNvRCS7ZM2KHuJnyO5sbKatU3ebEpHskVVBf960cXR0OW/UHkn3UERERk1WBf2iYIfs+r2H0jwSEZHRk1VBP3V8PuXFubyy6+DQnUVEQiKrgt7MuGhuOdW7D+Cuo29EJDtkVdADLJlVxjtH2tihe8iKSJbIuqC/+MxyAF7eeSDNIxERGR1ZF/RzyouYVJLHc9sa0j0UEZFRkXVB33Pi1DNbG2jt0PH0IhJ+WRf0AFedO5lj7V08t60x3UMRERlxQwa9md1rZvVmtnGA7WZmd5nZdjN73cwWJ2xbZmZbg223p3Lgw3HhnImMy4/y8Lp96R6KiMiIS2ZFfx+wbJDtVwPzgp+VwI8AzCwC3B1snw+sMLP5wxlsquRGc5hdUcyjr9cp7EUk9IYMend/FhjsEJXrgH/xuBeBUjObAiwBtrv7DndvBx4M+o4Jn710LgBPbHonzSMRERlZqajRTwP2JjyvCdoGah8Tlp03mXH5UX67oU4nT4lIqKUi6K2fNh+kvf83MVtpZtVmVt3QMDqHPp5fOR6AJ97Qql5EwisVQV8DTE94XgnUDtLeL3df5e5V7l5VUVGRgmEN7SefeA8Af79m86h8nohIOqQi6B8BPhEcfXMhcNjd64BXgHlmNtvMcoHlQd8xoyA3wjXnT2b3/mPs0iURRCSkkjm88gHgBeBsM6sxs8+Y2c1mdnPQZQ2wA9gO/BPwVwDu3gncCvwO2Az8q7tvGoE5DMsXrjoHgD++qTNlRSSchryVoLuvGGK7A7cMsG0N8S+CMWvWxELmlBfx1Uc28aELpjCxOC/dQxIRSamsPDM2kZlx2xXzAPjw9/+U5tGIiKRe1gc9wEcWTAWg9nArh461p3k0IiKppaAnvqp/9L+9D4CFdz6p4+pFJFQU9IHzpo0//njtbt1qUETCQ0Gf4Pd/cykAN/70Zdo7u9M8GhGR1FDQJzhzUjH//QPzaOnoYvmqF9I9HBGRlFDQ9/LXwRE4r+45xNf+35g77F9E5JQp6HsxM17+8gcA+Nmfd/HvvvUH2jp1JyoRyVwK+n5MKsnnvk/Hr4Oz90ALtz2wLs0jEhE5fQr6ASw9exL3/OW7AXh809us3T3YJflFRMYuBf0glp03mXs/VQXAf/jRC7qZuIhkJAX9EC4/5wwunjsRgHP+7nFa2hX2IpJZFPRJuP+/XMic8iIA3vWVx/n5C7vSOh4RkVOhoE/SU5+/lLxo/H+uv3t4E5/+2cu6VIKIZAQFfZLMjK3/5+rjNfuntzbwyZ+9Qne3wl5ExjYF/Sm6/Jwz+MZHzwfg2TcbmPOlNWx7pynNoxIRGZiC/jSsWDLj+HVxAK78zrNsr29SKUdExiQbi+FUVVXl1dXV6R5GUj71s5d5ZuuJ2xBWzSzjf1x1NhfOmZjGUYlItjGzte5e1d82reiH6b5PLzl+LXuA6t0HWb7qRd6oPZLGUYmInKCgT4Hzpo1n1zev5drzpxxvu+au57jll6+qnCMiaZdU0JvZMjPbambbzez2frZ/wczWBT8bzazLzCYE23aZ2YZgW2bUY07T3R9fzM5vXHP8+W831DH7jjUcPtaRxlGJSLYbskZvZhHgTeBKoAZ4BVjh7m8M0P/DwF+7++XB811Albs3JjuoTKrRD6SptYPz/9cTJ7V9b/lCpowvYMnsCWkalYiE1XBr9EuA7e6+w93bgQeB6wbpvwJ44NSHGS4l+TF2fuMarl807XjbbQ+u42M/foHvP7VNJR0RGTXJrOhvAJa5+03B8xuB97r7rf30LSS+6j/T3Q8EbTuBg4ADP3b3VQN8zkpgJcCMGTPevXv37tOe1Fj029fruOX+V09qWzSjlO98bCGzgssriIicrsFW9NFkXt9P20DfDh8G/twT8oFL3L3WzCYBT5rZFnd/ts8bxr8AVkG8dJPEuDLKtRdM4doLrmVnYzOX/eMzALy25xBL//EZlsyeQEVxHt9dvpCubic/FknvYEUkVJIJ+hpgesLzSqB2gL7L6VW2cffa4He9ma0mXgrqE/TZYnZ5Ebu+eS2HjrWz8M4nAXh5Z/x78bcb6gB4z6wy7v74YiaV5KdtnCISHsmUbqLEd8Z+ANhHfGfsf3L3Tb36jQd2AtPdvTloKwJy3L0pePwkcKe7Pz7YZ4ZhZ2yyurudJ954h5t/sbbPtgvnTOALV51NYW6UmRMLKcxN5ntZRLLRYKWbpM6MNbNrgO8CEeBed/+6md0M4O73BH0+RbyWvzzhdXOA1cHTKHC/u399qM/LpqBP1NbZxa7GY1z13YH/4PnkRTMZXxDjtivOIpLTX1VNRLLRsIN+tGVr0Ceqb2rltT2H+K8/77vS7+2f//MS3j+vHDMFv0i2UtCHQHe3s2ZjHbfe/9qg/T66aBr/8B8XkGMo+EWyiII+hFo7uli/9xAPr6/l/pf2DNhv5sRC3jt7At+6YcEojk5ERpuCPgtse6eJL/76dV7bcyip/ktmT6A4L8pZZ5Twt8vO1upfJMMp6LNQd7fz8xd3s7Oxmfue35X0665fNI3PvG827nB+5fiRG6CIpJSCXgDo7OrmUEsHb77dxN8/tpmN+5K/lPJ7ZpXx9evPZ+bEQvKiOqFLZKxR0MugWju6WLv7IHsOHOOOhzac1nucP208//Pad3F+5Xgd7y+SBgp6OS3tnd10u7N7/+DH9vcWzTE6g5umj8uP8p5ZE7h+8TQeenUfn7x4FpeeVcHTW+s5d8o4Jo3T2b8iqaCgl5Rqbuuk9lALXe48vK6WyePyufPRN+jqHv6/pVsum0tBLMKV8ydz9uSS4+2tHV1EcoxYRPfKEemPgl5GVXe389SWejbUHGJicR679jfzq+oamto6U/YZC6eX8sVlZ3Px3HL27D9GZVkBjUfb9BeCZC0FvYxJR1o7aOvoZlt9ExXFeVz5nWfJi+bQ1tmdkvf/0jXn8Oft+7nxwpksmF5KQ1MbB5rbKcqLcP0Pn+ePX1hKaWEurR1dnKEvCMlwCnrJWK0dXcQiOURyjJ2Nzfxq7V6e2PQO2+qPntTvY1WV/Gt1Tco+t6Ikj+dvv/x4qcjd2dHYTH4swoTCXB5Zv49/v2ga6/ceZm5FEeMKYiorSVop6CVrtHZ0sb3+KE9vqeeh1/Yxt6KI32+uH5XPjkWMFUtmsHhGGe1d3dQdauU7v3+Tz10xj89dcdaojEGyl4JepB/tnd3kRnNobuukpaOLkvwoOxqaqT3Uws7GZl7ccYDmtk5e2LE/JZ9Xkhelqa2T8QUxDrecuGH8OZNL2PJ2U5/+3/jo+Xx08TRiOTk0Hm0D4GhbJ1NLC066OU13t9Pc3km3x49ySsVZzu7O4ZYOSgtzh/1eMjoU9CIjYPf+Zs4Yl0+OGTkGj6yv5feb3+GlHfEbyZw5qZgb3l3Jd558k/3N7Snb9zCUyrICag629Gn/yIKpFMQinDdtHItmlDGhKBcnfiJdRUkev3xxD3sPHiOSY9z3/C7cYfVfXcyiGWW4O+6wveEo08sKKcgd+KS5nsNydae00aWgFxlD2jq7cAd36HKnoamN4rwoJflRDh5r5+afr+XSsyq46w/bgZNX/JNK8jjQ3H78PIV0mlZawNTSfF7ZdZCL505k4fRS5lYU8/l/Ww/Ez6a+flElkRx4/1kVlOTHKMqN8PvN9Vxy5kQMG/ALo/ZQC13dzvQJhaM5pYymoBfJEh1d3bjDoZZ2Dh/rYFPtEVo7unhqSz179h+j8Wgb+5vbB3z9nPIidjQ29ykvjZQp4/OpO9yadP8FleN575yJ/OLF3Rxr7wJg8YxS9hxoYW5FES8Ft+W8/JxJbKtv4nvLF1F/pI2Lz5zIvoMt5EZzyI3kUFlWQN3hVuoOt3Du1PG0dXZTmBvhT9sbWXpWxYDlr65up6Orm2PtXRxobmf6hIIxc0kQBb2IDFt3t2MGNQdbeGZrPUV5UeZWFFN3uIWGpjbW7T3MhXMmcO7U8fzTczt4bGMdf1E1nYde3ZfScyhGy8LppazbO/TVYD8anPUNMGNCIXsOHBuwHw6Tx+czf+o4/vZXrzO1tIBVn6ii8Wj80N9zJpcwc2LRaY1XQS8iY5q791lF97TtPXCMiqBktXb3QfJjERbNKGV8QYy6Q/G/Blo6urj/pd2sfm0fV86fjBm8c6SVqeML2H0gvmM9N5pDe7Cf5Lxp407pon6jadPXrqIo79SvF6WgFxEZAZ1d3XR2O5EcY+vbTazZUEdZYS47Gptpau1g6dmTuPSsCtbuPsAruw7y9NZ6csz4i6rpbHm7iRyDf1sbP/+jvDiP268+hxveXXlaY0nFzcGXAd8jfnPwn7j7N3ttXwo8DOwMmh5y9zuTeW1/FPQiIqdmsKAf8u8DM4sAdwNXAjXAK2b2iLu/0avrc+7+odN8rYiIjJBkztleAmx39x3u3g48CFyX5PsP57UiIpICyQT9NGBvwvOaoK23i8xsvZk9ZmbnnuJrMbOVZlZtZtUNDQ1JDEtERJKRTND3d0Bp78L+q8BMd18AfB/4zSm8Nt7ovsrdq9y9qqKiIolhiYhIMpIJ+hpgesLzSqA2sYO7H3H3o8HjNUDMzMqTea2IiIysZIL+FWCemc02s1xgOfBIYgczm2zBQbBmtiR43/3JvFZEREbWkEfduHunmd0K/I74IZL3uvsmM7s52H4PcAPwWTPrBFqA5R4/brPf147QXEREpB86YUpEJAQy7sxYM2sAdp/my8uBxhQOJ53CMpewzAM0l7EoLPOA4c1lprv3eyTLmAz64TCz6oG+1TJNWOYSlnmA5jIWhWUeMHJz0U0uRURCTkEvIhJyYQz6VekeQAqFZS5hmQdoLmNRWOYBIzSX0NXoRUTkZGFc0YuISAIFvYhIyIUm6M1smZltNbPtZnZ7usfTHzO718zqzWxjQtsEM3vSzLYFv8sStt0RzGermV2V0P5uM9sQbLur5/IToziP6Wb2tJltNrNNZnZbBs8l38xeDq68usnMvpapcwnGEDGz18zs0UyeRzCOXcE41plZdabOx8xKzexXZrYl+G/molGfh7tn/A/xyyu8BcwBcoH1wPx0j6ufcb4fWAxsTGj7FnB78Ph24P8Gj+cH88gDZgfziwTbXgYuIn510MeAq0d5HlOAxcHjEuDNYLyZOBcDioPHMeAl4MJMnEswhr8B7gcezdR/Xwlz2QWU92rLuPkA/wzcFDzOBUpHex6j/n/eCP0PeRHwu4TndwB3pHtcA4x1FicH/VZgSvB4CrC1vzkQv17QRUGfLQntK4Afp3lODxO/i1hGzwUoJH7J7fdm4lyIXx32KeByTgR9xs0j4bN30TfoM2o+wDjit1i1dM4jLKWbpG9wMgad4e51AMHvSUH7QHOaFjzu3Z4WZjYLWER8JZyRcwnKHeuAeuBJd8/UuXwX+CLQndCWifPo4cATZrbWzFYGbZk2nzlAA/CzoKT2EzMrYpTnEZagT/oGJxlkoDmNmbmaWTHwa+Bz7n5ksK79tI2Zubh7l7svJL4iXmJm5w3SfUzOxcw+BNS7+9pkX9JPW9rn0csl7r4YuBq4xczeP0jfsTqfKPFy7Y/cfRHQTLxUM5ARmUdYgj6Tb3DyjplNAQh+1wftA82pJnjcu31UmVmMeMj/0t0fCpozci493P0Q8AywjMybyyXAR8xsF/F7M19uZr8g8+ZxnLvXBr/rgdXE70GdafOpAWqCvxIBfkU8+Ed1HmEJ+ky+wckjwCeDx58kXu/uaV9uZnlmNhuYB7wc/JnXZGYXBnvdP5HwmlERfO5Pgc3u/u2ETZk4lwozKw0eFwBXAFvIsLm4+x3uXunus4j/+/+Du/9lps2jh5kVmVlJz2Pgg8BGMmw+7v42sNfMzg6aPgC8MerzSMdOlhHa6XEN8aM/3gK+nO7xDDDGB4A6oIP4N/RngInEd6BtC35PSOj/5WA+W0nYww5UEf9H/xbwA3rt6BmFebyP+J+NrwPrgp9rMnQuFwCvBXPZCHwlaM+4uSSMYykndsZm5DyI17bXBz+bev6bzsT5AAuB6uDf2G+AstGehy6BICIScmEp3YiIyAAU9CIiIaegFxEJOQW9iEjIKehFREJOQS8iEnIKehGRkPv/IXO7tx50sLYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss_history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de7xVdZ3/8df7wJHDHYSjIneVNDVEPXJJS7KmQTTpwhhOaVmOv7SL+qvMLj/Hafo1Ns00ZRcdTaexQa3wWqFlJWnlDQgQEJWbckQFkauA3D7zx3cd2RzOORzgrLPPYb2fj8d+7L3XWnutz3ej+32+a33XWooIzMysuCrKXYCZmZWXg8DMrOAcBGZmBecgMDMrOAeBmVnBOQjMzArOQWD7RNI8SWPLuP1BkjZI6tDC6/24pD+15Dr3Ytv3S/pYObZtxdax3AVY+xQRx9W9lnQNcFREfDSv7UlaClwUEb/Ltv8C0C2v7ZVDRJxZ7hqsmNwjsLKTdMD/QXIgtPFAaIM1zEFg+0TSUknvkTQO+Arw4WxXzexsfk9JN0t6SdKLkr5Rtxsn2/3yZ0n/Iek14BpJR0r6g6RVkl6VNFlSr2z5nwKDgF9m27hS0hBJIamjpEmSpter7wpJ92WvO0n6N0kvSHpF0g2SOjezncdIelDSa5KekXRuybyzJP1V0jpJy7KeUd28uvo+KekF4A91u52yWlZLWiLpzJLPTJN0Ucl31NSyQyU9LGm9pN9J+qGk/2miHRMkzcpqXZT9u73571iy3DV162mkDQ9I+ky9dc+W9ME9fV/WdjkIbL9ExAPAN4GfRUS3iDghm/XfwDbgKOBE4L3ARSUfHQUsBg4B/j8g4F+Aw4G3AgOBa7JtnA+8ALwv28a/1ivjPuBoScNKpv09cFv2+lvAW4ARWT39gav31DZJXYEHs/UcApwH/EhS3W6x14ELgF7AWcAlkt5fbzWnZ+3525J2PwP0Bf4VuFmSGimhqWVvA54A+pC+p/ObaMdI4Fbgi1mt7wSWNtn4xttwG+l7qFv3scBg4NfN+L6sjXIQWIuTdChwJnB5RLweESuA/wAmlSy2PCK+HxHbImJTRCyMiAcj4o2IWAl8h/QDtEcRsRG4l+wHKguEY4D7sh/OfwCuiIjXImI9KbgmNba+EmcDSyPiv7I6ZwJ3AhOz7U6LiKciYkdEzAFub6Dma7LvYFP2/vmIuCkitpPCsh9waCPbb3BZSYOAU4CrI2JLRPyJFIaN+SRwS/b97oiIFyNiQTPa31Ab7gZGSBqczfsIcFdEvMEevi9ru7zPz/IwGKgEXir5Y7cCWFayTOlrJB0CXAe8A+ieLb96L7Z5G/DvwNdJvYF7ImJjtt4uwIySWgQ0Z7TRYGCUpDUl0zoCP81qHgVcCxwPHAR0An5Rbx3L6r1/ue5FVh80ftC7sWX7Aq9lAVi6nYGNrGcgMLWRec3xZhsiYr2kX5OC9FvZ88XZ7Ca/L2u7HATWEupfwnYZ8AbQNyK2NfMz/5JNGx4Rq7JdLD9oYvn6fgv0lTSC1DO4Ipv+KrAJOC4iXtzDOupbBvwxIv6mkfm3ZTWeGRGbJX2X9CNdKo/L+74EHCypS0kYNBYCkNpxZCPzXicFZZ3DGlimfhtuB/5R0sNAZ+Chku009X1ZG+VdQ9YSXgGGSKoAiIiXSD/M/y6ph6SK7GBwU7t6ugMbgDWS+pP2Z9ffxhGNfTgLnCnAt4GDSfuqiYgdwE3Af2S9AyT1l/S3ja2rxK+At0g6X1Jl9jhF0ltLan4tC4GRpJ5I7iLieWA66SD7QZLGAO9r4iM3AxdKenf2b9Ff0jHZvFnApKxtNTRvN85U0l//XycdG9qRTd/T92VtlIPAWkLd7pBVkmZmry8g7S6ZT9rFM4W0j7sx/wScBKwFfg3cVW/+vwBfk7RG0hcaWcdtwHuAX9TriXwJWAg8Jmkd8Dvg6D01Kjue8F7S7o/lpF013yLtAgK4FPi6pPWkg88/39M6W9BHgDHAKuAbwM9IvbDdRMQTwIWk4zRrgT+SfsgB/h+pt7Ca9G9wW0PrqLe+N0j/Pu8pXb4Z35e1UfKNaczaP0k/AxZExD+WuxZrf9wjMGuHsl0uR2a7esYBE4B7yl2XtU+5BYGkKklPZCebzJP0Tw0sI0nXSVooaY6kk/Kqx+wAcxgwjXRc5Trgkoj4a1krsnYrt11D2fjtrhGxQVIl8Cfgsoh4rGSZ8cBngfGkk2e+FxGjcinIzMwalFuPIJIN2dvK7FE/dSYAt2bLPgb0ktTUAUUzM2thuZ5HoHRtmRmk0/p/GBGP11ukP7uecFObTXup3nouJjtppWvXricfc8wxtKTNm2HePBgyBPr0adFVm5m1CTNmzHg1IqobmpdrEGSnxo9QunjY3ZKOj4i5JYs0dI2V3fZVRcSNwI0ANTU1MX369N0+tD/uuAPOOw/uuQdOOGHPy5uZtTeSnm9sXquMGoqINaQDW+Pqzapl1zMiB5DGH7eqefOgogJauKNhZtYu5DlqqFo7LyPcmXTySf0LXd0HXJCNHhoNrM3OSm1VM2bAscdCJ5/2YmYFlOeuoX7Af2fHCSqAn0fEryR9CiAibiCdqj6edNbnRtLZj61uzhw444xybNnMrPxyC4LssrwnNjD9hpLXAXw6rxqaY/VqePFFeNvbylmFmeVp69at1NbWsnnz5nKXkruqqioGDBhAZWVlsz9T+KuPzs0OXR9/fHnrMLP81NbW0r17d4YMGULj9wFq/yKCVatWUVtby9ChQ5v9ucJfYmL+/PR87LHlrcPM8rN582b69OlzQIcAgCT69Omz1z2fwgfBc89BVRUMbOpq7mbW7h3oIVBnX9rpIHgOjjoqDR81Myuiwv/8PfssDBu25+XMzPbVmjVr+NGPfrTXnxs/fjxr1qzZ84L7qdBBsH07LF7sIDCzfDUWBNu3b2/yc1OnTqVXr155lfWmQo8aWrQItmzxGcVmlq+rrrqKRYsWMWLECCorK+nWrRv9+vVj1qxZzJ8/n/e///0sW7aMzZs3c9lll3HxxRcDMGTIEKZPn86GDRs488wzOe200/jLX/5C//79uffee+ncuXOL1FfoIJg3Lz176KhZgVx+Ocya1bLrHDECvvvdRmdfe+21zJ07l1mzZjFt2jTOOuss5s6d++YQz1tuuYWDDz6YTZs2ccopp/ChD32IPvWugPncc89x++23c9NNN3Huuedy55138tGPfrRFyi90ECxalJ6PPLK8dZhZsYwcOXKXcf7XXXcdd999NwDLli3jueee2y0Ihg4dyogRIwA4+eSTWbp0aYvVU+ggmDULDjsMevcudyVm1mqa+Mu9tXTt2vXN19OmTeN3v/sdjz76KF26dGHs2LENngfQqeRiaB06dGDTpk0tVk+hDxYvXQpHHw0FGV5sZmXSvXt31q9f3+C8tWvX0rt3b7p06cKCBQt47LHHGlwuT4XuETz/PLzrXeWuwswOdH369OHUU0/l+OOPp3Pnzhx66KFvzhs3bhw33HADw4cP5+ijj2b06NGtXl9hg2D9eqitTSeTmZnl7bbbbmtweqdOnbj//vsbnFd3HKBv377Mnbvznl5f+MIXWrS2wu4aqjtQ7GsMmVnRFT4IPGLIzIqusEGwcGF6dhCYWdEVNggWLYLqaujRo9yVmJmVV2GDYOFCHyg2M4MCB8GiRd4tZGYGBQ2CN96AZcvcIzCztqlbt24ALF++nIkTJza4zNixY5k+fXqLbK+QQbBkCUS4R2Bmbdvhhx/OlClTct9OIU8oqxsx5B6BmbWGL33pSwwePJhLL70UgGuuuQZJPPzww6xevZqtW7fyjW98gwkTJuzyuaVLl3L22Wczd+5cNm3axIUXXsj8+fN561vf2qLXGipkEPgcArPiKsNVqJk0aRKXX375m0Hw85//nAceeIArrriCHj168OqrrzJ69GjOOeecRu85fP3119OlSxfmzJnDnDlzOOmkk1qs/kIGwcKFadho377lrsTMiuDEE09kxYoVLF++nJUrV9K7d2/69evHFVdcwcMPP0xFRQUvvvgir7zyCocddliD63j44Yf53Oc+B8Dw4cMZPnx4i9VXyCCoGzHkq46aFU+5rkI9ceJEpkyZwssvv8ykSZOYPHkyK1euZMaMGVRWVjJkyJAGLz9dqrHewv7K7WCxpIGSHpL0tKR5ki5rYJmxktZKmpU9rs6rnlI+h8DMWtukSZO44447mDJlChMnTmTt2rUccsghVFZW8tBDD/H88883+fl3vvOdTJ48GYC5c+cyZ86cFqstzx7BNuDzETFTUndghqQHI2J+veUeiYizc6xj16K2pfsQfOhDrbVFMzM47rjjWL9+Pf3796dfv3585CMf4X3vex81NTWMGDGCY/Zw8/RLLrmECy+8kOHDhzNixAhGjhzZYrXlFgQR8RLwUvZ6vaSngf5A/SBoVcuWwdat7hGYWet76qmn3nzdt29fHn300QaX27BhA5BuXl93+enOnTtzxx135FJXq5xHIGkIcCLweAOzx0iaLel+Scc18vmLJU2XNH3lypX7VYtHDJmZ7Sr3IJDUDbgTuDwi1tWbPRMYHBEnAN8H7mloHRFxY0TURERNdXX1ftXjcwjMzHaVaxBIqiSFwOSIuKv+/IhYFxEbstdTgUpJuQ7qXLQIOnWCww/Pcytm1tZERLlLaBX70s48Rw0JuBl4OiK+08gyh2XLIWlkVs+qvGoCWLwYhg6FikJeXMOsmKqqqli1atUBHwYRwapVq6iqqtqrz+U5auhU4HzgKUl15/F9BRgEEBE3ABOBSyRtAzYBkyLnf6kXXoDBg/Pcgpm1NQMGDKC2tpb9PcbYHlRVVTFgwIC9+kyeo4b+BDR59kNE/AD4QV41NOSFF+DEE1tzi2ZWbpWVlQwdOrTcZbRZhdpBsmEDrFgBgwaVuxIzs7ajUEFQN4T3bW8rbx1mZm1JoYJg9uz0PGJEeeswM2tLChUEs2ZBr17eNWRmVqpQQTB7Npxwgq86amZWqjBBsH07zJmTgsDMzHYqTBAsWgQbN/r4gJlZfYUJgrpb07lHYGa2q8IEwTvetobbvr6Q445ouRs+m5kdCAoTBP2e+i3nXT2MTsuXlLsUM7M2pTBBQJcu6XnjxvLWYWbWxjgIzMwKzkFgZlZwDgIzs4JzEJiZFZyDwMys4BwEZmYF5yAwMyu44gRBZSV06OAgMDOrpzhBIKVegYPAzGwXxQkCSEHw+uvlrsLMrE0pXhC4R2BmtgsHgZlZwTkIzMwKzkFgZlZwuQWBpIGSHpL0tKR5ki5rYBlJuk7SQklzJJ2UVz2Ag8DMrAEdc1z3NuDzETFTUndghqQHI2J+yTJnAsOyxyjg+uw5Hw4CM7Pd5NYjiIiXImJm9no98DTQv95iE4BbI3kM6CWpX141OQjMzHbXKscIJA0BTgQerzerP7Cs5H0tu4cFki6WNF3S9JUrV+57IQ4CM7Pd5B4EkroBdwKXR8S6+rMb+EjsNiHixoioiYia6urqfS/GQWBmtptcg0BSJSkEJkfEXQ0sUgsMLHk/AFieW0Fdu6Yzi3fsyG0TZmbtTZ6jhgTcDDwdEd9pZLH7gAuy0UOjgbUR8VJeNdGzJ0T4MhNmZiXyHDV0KnA+8JSkWdm0rwCDACLiBmAqMB5YCGwELsyxHujRIz2vXQvdu+e6KTOz9iK3IIiIP9HwMYDSZQL4dF417KZnz/S8di0MGNBqmzUza8uKdWZxaRCYmRngIDAzKzwHgZlZwRUzCNbVP53BzKy4ihUEpaOGzMwMKFoQdOsGFRUOAjOzEsUKAin1ChwEZmZvKlYQQDpO4CAwM3tTMYPAB4vNzN5UzCBwj8DM7E0OAjOzgiteEPhgsZnZLooXBO4RmJntorhBELvdCM3MrJCKFwR9+sC2bbB+fbkrMTNrE4oXBIcckp5XrChvHWZmbUTxgqC6Oj2vXFneOszM2ojiBYF7BGZmuyheELhHYGa2i+IGgXsEZmZAEYOgc+d0OWr3CMzMgCIGAaTjBO4RmJkBRQ2C6mr3CMzMMsUMgkMOcRCYmWWKGwSvvFLuKszM2oTcgkDSLZJWSJrbyPyxktZKmpU9rs6rlt0MGAAvvwxbtrTaJs3M2qo8ewQ/AcbtYZlHImJE9vh6jrXsatCgdNG52tpW26SZWVuVWxBExMPAa3mtf78MHpyen3++vHWYmbUB5T5GMEbSbEn3SzqusYUkXSxpuqTpK1viIK+DwMzsTc0KAkmXSeqh5GZJMyW9dz+3PRMYHBEnAN8H7mlswYi4MSJqIqKmuu7M4P0xcGB6fuGF/V+XmVk719wewSciYh3wXqAauBC4dn82HBHrImJD9noqUCmp7/6ss9k6dYJ+/dwjMDOj+UGg7Hk88F8RMbtk2j6RdJgkZa9HZrWs2p917pXBgx0EZmZAx2YuN0PSb4GhwJcldQd2NPUBSbcDY4G+kmqBfwQqASLiBmAicImkbcAmYFJEK94/cvBgmDGj1TZnZtZWNTcIPgmMABZHxEZJB5N2DzUqIs7bw/wfAD9o5vZb3qBBcPfdsGMHVJT7mLmZWfk09xdwDPBMRKyR9FHga8Da/MpqBYMHpxPKfPE5Myu45gbB9cBGSScAVwLPA7fmVlVrGDIkPS9ZUtYyzMzKrblBsC3bfz8B+F5EfA/onl9ZrWDYsPT87LPlrcPMrMyae4xgvaQvA+cD75DUgezAb7s1dChUVsIzz5S7EjOzsmpuj+DDwBuk8wleBvoD386tqtZQWQlHHQULFpS7EjOzsmpWEGQ//pOBnpLOBjZHRPs+RgBwzDEOAjMrvOZeYuJc4Ang74BzgcclTcyzsFZx9NGwcCFs3VruSszMyqa5xwi+CpwSESsAJFUDvwOm5FVYqxg+PIXAnDlw8snlrsbMrCyae4ygoi4EMqv24rNt19vfnp6feKK8dZiZlVFzf8wfkPQbSR+X9HHg18DU/MpqJYMGwcEHw5NPlrsSM7OyadauoYj4oqQPAaeSLjZ3Y0TcnWtlrUGC006DP/+53JWYmZVNc48REBF3AnfmWEt5jBoF990Hq1dD797lrsbMrNU1uWtI0npJ6xp4rJe0rrWKzNWoUen5scfKW4eZWZk0GQQR0T0iejTw6B4RPVqryFyNGQNVVXD//eWuxMysLNr/yJ/91aULjB0LDz5Y7krMzMrCQQApCBYs8CWpzayQHASQggDg978vaxlmZuXgIACoqUk3s//FL8pdiZlZq3MQAHToACNHpltX+v4EZlYwDoI6H/94ev7lL8tahplZa3MQ1DnnHKiu9uUmzKxwHAR1Kirggx+EO++EdQfGuXJmZs3hICj1sY/Btm0weXK5KzEzazUOglKjR8OIEXDDDRBR7mrMzFqFg6CUBJ/9bLpRzV13lbsaM7NWkVsQSLpF0gpJcxuZL0nXSVooaY6kk/KqZa+cf346aDxxImzfXu5qzMxyl2eP4CfAuCbmnwkMyx4XA9fnWEvzVVbCl76UXh9ySHlrMTNrBbkFQUQ8DLzWxCITgFsjeQzoJalfXvXslSuuSM+vvQbz5pW3FjOznJXzGEF/YFnJ+9ps2m4kXSxpuqTpK1euzL+yioqd5xMcf7wPHJvZAa2cQaAGpjX4ixsRN0ZETUTUVFdX51xWpqZm58XoPvvZ1tmmmVkZlDMIaoGBJe8HAMvLVEvDfv97OOUU+OEPYdmyPS9vZtYOlTMI7gMuyEYPjQbWRsRLZaxndxUVcH12DHvQIHj66fLWY2aWgzyHj94OPAocLalW0iclfUrSp7JFpgKLgYXATcCledWyX04+OfUIAI491tciMrMDTse8VhwR5+1hfgCfzmv7LerSS6F/f3j/+9PlqtesgZ49y12VmVmL8JnFzTVhAlx2WXrdqxc8/nh56zEzayEOgr3x3e/u3E00ejR89aswe3Z5azIz208Ogr116aUwbVp6/c1vpovU/fGPZS3JzGx/OAj2xemnw/r1O9+PHQuXX+4Tz8ysXXIQ7Ktu3dIP/69+ld5/73tpuOnVV8PWreWtzcxsLzgI9tdZZ8Hrr8O73pXe//M/w0EHwfjx5a3LzKyZHAQtoUsX+MMfYPlyOPfcNO3++9P9DSTYsqW89ZmZNcFB0JL69YOf/QyWLIG3vnXn9GOO2RkKM2ak8xDMzNoIB0EehgyB+fPTsYKbb07v69TUQO/eqQdhZtYGOAjy1LEjfOIT6Uf/2Wd3nffud6cewje/6eGnZlZWDoLWMmxYGmUUke6HfMopafpXv5qGn9btOurVKw1N9VBUM2slDoJy+MAH4Ikn0mijX/wi7Sqqs3Yt9OiRhqL+3d/BnXfC4sUekmpmuXEQlFOXLjBxYrolZgQsWgT33JPOVgaYMiXNP/LINCS1rtdw5ZUpHNxrMLMWoGhnPyY1NTUxffr0cpfROtasgV/+Ei64oOH5Rx0FCxfufP/Tn0LnzjBwYLpKqplZRtKMiKhpaJ57BG1Zr15w/vk7jy3s2JHOVbjoojSv/v2bzz8/9SBGjYI+fXb2II44Ah56KN1Yx7uYzKweB0F7IqVzFW66CVavTj2GupB46in42td2LvvaaztfL1kCZ5yRbqxTuovphz9Mz5//PCxYkG7HWRcUEbBtW+u2z8zKwruGDnSLF8ODD8Ldd8NvfrNv6/jGN3aGzNKlaZ1vfzt06pSmRaRAMbM2q6ldQw6Cotq2LfUAVqxIl8OYNm3/zmc4/fQ04mn8eHjLW9KuqfPOSyfUSenubh07OjDMysRBYPtu+/Z0Mty0aVD3vd9yS/qhX7Vq79d31FEwaFA6vvHhD6cwGjgQbrstBckVV6QD3Y8+Ch/8YDoWsnVrGmHVvXsaVmtme81BYPnbtAk2bky38Jw7N/14P/IIzJyZfvQPPRReeaVlttW9O3zsY1BVBTfeCOvWpemnnw5nnpnqqKhItxbt0iUdL9m+Haqr0zES78qyAnIQWNuzdWv6ge7cOQXEI4+k97ffnkY+rV+fRjndc0/Dn6+oSKOo9kZl5a6jpkaNgpdeSj2PRx5JJ/hNmADf/34Kk3POSQfl161LB9t79oQHHoAxY+Bb30rrK7V9O3TokK42K6X5mzenwDIrMweBHXh27Ei9kIUL002CunVLZ2svWAAvvghTp6Yf5A98IPUgDj00jaz6z/9suRqGDEnr/etf93yp8QED0n2uV6+GU09NZ4/ffXe6Gu3tt8PRR6fey6ZN6RjLunUppI4+OrV1y5Y0r2vXFIIdOqSHWTM5CMwas3172nVVVZV6JwsWpPcHHQTPPJNGR1133c5huvfck3oVY8emH+R16+D3v9/zdjp23PfhuAcd1HDQnH56CpTVq1PvaciQtI3+/WHDhhQaxx6bjr38+MfpMx//eLoC7nPPwfPPp2tePflkCqeqqtS2005Lnx04MA097tEjjRSrrU3fyfjx8MUvwrhx6ZhOp04prDZsSL2qgQObPpazfHkaBu3dc63KQWBWDvWPRWzZki4jsmVL+nE+6qj0wz1/fpr++uspiCor0w/rjBnpciObNqX1bNyYDtD37p3OOO/WLf3oPv10Wn/fvjuPxbz6annaXGfUqDQQYPnyVNdZZ6XAWLIkDWeG1I4zzkihcdRRqeaZM1ObBwxIAwUWL07TO3RI03r0SJ+vrYV/+Af485/T93PppSkI+/ZN4XzrrSnMxo1LuxwPPjh9dvJk+OQnU3gefnhafuvWFLbbtqXArvt3q/ttrPs33L49LdtOd/U5CMyKpq4HU3eMoqIivV+zJk1bty6F0csvp9BZsQLmzNl50cOuXeHee9OP7U9+kqZt3pyGBG/alHa/PfNM+qGuqoLhw9P9u994I22/X7+0a6vOoEE755UOGujZM22zLanblVh6+ZZSvXun3XxVVen73LYtfX8VFTtDuc6YMSlsfvnL9P6ii3b2zqqrd14doKoqXUNsyRI47LD0vc6cmY5ZHXtsGkXXv396X9Pgb/kelS0IJI0Dvgd0AH4cEdfWmz8WuBdYkk26KyK+3tQ6HQRmB6A33kihtH59+gt827adQfXGG3DccSmIDj88hdCRR6bg+sMfYN68dHmVV1/defxkx4701/3q1Wldf/xj6kls355+5Hv0SCdHfvrT6RjP9u3wpz+lwHrhhTRQoKIi7Qo86CB4xztST+6RR9LQ6QED0vsXXkjrW7w4/eDX9cTqAq6lg+7KK9NAhX1QliCQ1AF4FvgboBZ4EjgvIuaXLDMW+EJEnN3c9ToIzKxd2rp159DmysoUcD17pp7ali0p4Dp0SD2mww9Pu/5WrUrHcA49NE3r1y+F2D5oKgg67lfDmjYSWBgRi7Mi7gAmAPOb/JSZ2YGosjL98NepO9YgpeMnRxyR3g8evHOZXr1S7ydneZ6m2R9YVvK+NptW3xhJsyXdL+m4hlYk6WJJ0yVNX1n/iptmZrZf8gyChsaG1d8PNRMYHBEnAN8HGjx7KCJujIiaiKiprq5u4TLNzIotzyCoBQaWvB8ALC9dICLWRcSG7PVUoFJS3xxrMjOzevIMgieBYZKGSjoImATcV7qApMOkNEhX0sisnn24kpmZme2r3A4WR8Q2SZ8BfkMaPnpLRMyT9Kls/g3AROASSduATcCkaG8nNpiZtXM+oczMrAB8z2IzM2uUg8DMrOAcBGZmBecgMDMrOAeBmVnBOQjMzArOQWBmVnAOAjOzgnMQmJkVnIPAzKzgHARmZgXnIDAzKzgHgZlZwTkIzMwKzkFgZlZwDgIzs4JzEJiZFZyDwMys4BwEZmYF5yAwMys4B4GZWcE5CMzMCs5BYGZWcA4CM7OCcxCYmRVcrkEgaZykZyQtlHRVA/Ml6bps/hxJJ+VZj5mZ7S63IJDUAfghcCZwLHCepGPrLXYmMCx7XAxcn1c9ZmbWsDx7BCOBhRGxOCK2AHcAE+otMwG4NZLHgF6S+uVYk5mZ1ZNnEPQHlpW8r82m7e0ySLpY0nRJ01euXNnihZqZFVmeQaAGpsU+LENE3BgRNRFRU11d3SLFmZlZkmcQ1AIDS94PAJbvwzJmZpajPIPgSWCYpKGSDgImAffVW+Y+4IJs9NBoYG1EvJRjTWZmVk/HvFYcEZFTnpgAAAamSURBVNskfQb4DdABuCUi5kn6VDb/BmAqMB5YCGwELsyrHjMza1huQQAQEVNJP/al024oeR3Ap/OswczMmuYzi83MCs5BYGZWcA4CM7OCcxCYmRWcg8DMrOAcBGZmBecgMDMrOAeBmVnBKZ3T1X5IWgk8v48f7wu82oLllJPb0jYdKG05UNoBbkudwRHR4FU7210Q7A9J0yOiptx1tAS3pW06UNpyoLQD3Jbm8K4hM7OCcxCYmRVc0YLgxnIX0ILclrbpQGnLgdIOcFv2qFDHCMzMbHdF6xGYmVk9DgIzs4IrTBBIGifpGUkLJV1V7noaIukWSSskzS2ZdrCkByU9lz33Lpn35aw9z0j625LpJ0t6Kpt3nSS1cjsGSnpI0tOS5km6rB23pUrSE5JmZ235p/balqyGDpL+KulX7bwdS7MaZkma3s7b0kvSFEkLsv9nxrR6WyLigH+QbpW5CDgCOAiYDRxb7roaqPOdwEnA3JJp/wpclb2+CvhW9vrYrB2dgKFZ+zpk854AxgAC7gfObOV29ANOyl53B57N6m2PbRHQLXtdCTwOjG6Pbclq+L/AbcCv2ut/X1kNS4G+9aa117b8N3BR9vogoFdrt6VVG1yuR/bl/Kbk/ZeBL5e7rkZqHcKuQfAM0C973Q94pqE2kO4NPSZbZkHJ9POA/yxzm+4F/qa9twXoAswERrXHtgADgN8DZ7AzCNpdO7LtLmX3IGh3bQF6AEvIBu6Uqy1F2TXUH1hW8r42m9YeHBoRLwFkz4dk0xtrU//sdf3pZSFpCHAi6S/pdtmWbHfKLGAF8GBEtNe2fBe4EthRMq09tgMggN9KmiHp4mxae2zLEcBK4L+yXXY/ltSVVm5LUYKgoX1l7X3cbGNtajNtldQNuBO4PCLWNbVoA9PaTFsiYntEjCD9RT1S0vFNLN4m2yLpbGBFRMxo7kcamFb2dpQ4NSJOAs4EPi3pnU0s25bb0pG0O/j6iDgReJ20K6gxubSlKEFQCwwseT8AWF6mWvbWK5L6AWTPK7LpjbWpNntdf3qrklRJCoHJEXFXNrldtqVORKwBpgHjaH9tORU4R9JS4A7gDEn/Q/trBwARsTx7XgHcDYykfbalFqjNepkAU0jB0KptKUoQPAkMkzRU0kHAJOC+MtfUXPcBH8tef4y0v71u+iRJnSQNBYYBT2TdyPWSRmejBi4o+UyryLZ7M/B0RHynZFZ7bEu1pF7Z687Ae4AFtLO2RMSXI2JARAwh/ff/h4j4aHtrB4CkrpK6170G3gvMpR22JSJeBpZJOjqb9G5gPq3dltY+yFOuBzCeNHplEfDVctfTSI23Ay8BW0kJ/0mgD+kA33PZ88Ely381a88zlIwQAGpI/2MsAn5AvQNRrdCO00jd0jnArOwxvp22ZTjw16wtc4Grs+ntri0ldYxl58HidtcO0n712dljXt3/z+2xLVkNI4Dp2X9j9wC9W7stvsSEmVnBFWXXkJmZNcJBYGZWcA4CM7OCcxCYmRWcg8DMrOAcBFY4kv6SPQ+R9PctvO6vNLQts7bMw0etsCSNBb4QEWfvxWc6RMT2JuZviIhuLVGfWWtxj8AKR9KG7OW1wDuya9pfkV1c7tuSnpQ0R9L/yZYfq3R/hduAp7Jp92QXPJtXd9EzSdcCnbP1TS7dlpJvS5qbXTP+wyXrnlZyPfrJddeRl3StpPlZLf/Wmt+RFUvHchdgVkZXUdIjyH7Q10bEKZI6AX+W9Nts2ZHA8RGxJHv/iYh4LbvsxJOS7oyIqyR9JtIF6ur7IOkM0hOAvtlnHs7mnQgcR7o2zJ+BUyXNBz4AHBMRUXeZC7M8uEdgttN7gQuyS04/TjrNf1g274mSEAD4nKTZwGOki4ANo2mnAbdHupLpK8AfgVNK1l0bETtIl+MYAqwDNgM/lvRBYON+t86sEQ4Cs50EfDYiRmSPoRFR1yN4/c2F0rGF9wBjIuIE0rWIqpqx7sa8UfJ6O9AxIraReiF3Au8HHtirlpjtBQeBFdl60q006/wGuCS7hDaS3pJd3bK+nsDqiNgo6RjSrSvrbK37fD0PAx/OjkNUk25L+kRjhWX3cugZEVOBy0m7lcxy4WMEVmRzgG3ZLp6fAN8j7ZaZmR2wXUn6a7y+B4BPSZpDugLkYyXzbgTmSJoZER8pmX436ZaCs0lXZr0yIl7OgqQh3YF7JVWRehNX7FsTzfbMw0fNzArOu4bMzArOQWBmVnAOAjOzgnMQmJkVnIPAzKzgHARmZgXnIDAzK7j/BXSiZtXbuPEFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss_history_valid = compute_loss_history(w_history,test_X[0:1000,:],test_y[0:1000],reg)\n",
    "\n",
    "plt.plot(loss_history, color='r')\n",
    "plt.plot(loss_history_valid, color='b') \n",
    "plt.ylim(0,5)\n",
    "plt.xlabel('iterations')\n",
    "plt.ylabel('loss')\n",
    "plt.title('iterative learning curve')\n",
    "plt.legend(['train', 'valid'])\n",
    "plt.ylim(-0.2,3)\n",
    "plt.show()  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练集的准确性： 0.82795\n",
      "测试集的准确性： 0.0997\n"
     ]
    }
   ],
   "source": [
    "print(\"训练集的准确性：\",getAccuracy(w,train_X,train_y))\n",
    "print(\"测试集的准确性：\",getAccuracy(w,test_X,test_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.7.4 随机梯度下降法\n",
    "\n",
    "批梯度下降法每次迭代只用一小批样本，而随机梯度下降法更加极端，每次迭代只用一个样本。因此，要执行随机梯度下降，只要将上述代码中调用批梯度下降法前将批大小batchsize修改为1，即每次只用一个样本更新模型参数，为了节省训练时间，将epochs改为2次："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w:  [[ 3.43980083e-01 -3.82689114e-01 -7.55376465e-02 ... -4.76886905e-02\n",
      "  -4.25716194e-01 -1.09449897e+00]\n",
      " [-1.86327347e-04 -8.12335257e-06 -1.44934912e-04 ... -1.14782815e-06\n",
      "  -5.92479477e-05 -5.37628266e-06]\n",
      " [ 7.51956062e-04 -7.79895436e-07 -5.96445960e-06 ... -1.39582421e-05\n",
      "  -6.60711501e-04 -2.38648043e-05]\n",
      " ...\n",
      " [-4.42026477e-02 -7.47829152e-03  7.41820965e-02 ... -8.84571836e-03\n",
      "  -7.05771139e-02 -3.17087510e-03]\n",
      " [-2.04871398e-02 -9.44509010e-04  2.08531136e-02 ... -2.77501709e-03\n",
      "  -3.71061063e-02  4.02894187e-03]\n",
      " [-8.15255742e-04 -5.30418756e-04  4.76885923e-03 ... -7.14505216e-04\n",
      "  -4.01528037e-03  1.51593039e-03]]\n"
     ]
    }
   ],
   "source": [
    "batchsize=1\n",
    "epochs = 2\n",
    "w = np.zeros([train_X.shape[1]+1,len(np.unique(train_y))])   \n",
    "w_history = batch_gradient_descent_softmax(w,train_X,train_y,epochs,batchsize,\n",
    "                                                shuffle,reg,alpha,gamma)\n",
    "w = w_history[-1]\n",
    "print(\"w: \",w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练集的准确性： 0.8041166666666667\n",
      "测试集的准确性： 0.785\n"
     ]
    }
   ],
   "source": [
    "print(\"训练集的准确性：\",getAccuracy(w,train_X,train_y))\n",
    "print(\"测试集的准确性：\",getAccuracy(w,test_X,y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
